filebeat: log when multiline reader truncates messages

mauri870 commented 1 week ago

Proposed commit message

While investigating an SDH, I noticed that although we add the truncated label to log fields, there is no feedback explaining why the messages were truncated.

This PR adds a log statement to provide insight whenever multiline messages are truncated.

Checklist

[x] My code follows the style guidelines of this project
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have made corresponding change to the default configuration files
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ] Is it okay for this to be a Warn level log? Should it be Debug?

How to test this PR locally

Create a log file with multiline logs whose line count or max bytes exceeds the limit.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - '/tmp/file.log'
  multiline:
    pattern: '^LINESTART$"),'
    match: after
    negate: true
    max_bytes: 104857600
    max_lines: 500
  index: "filebeat-multiline"

Related issues

-

Logs

{"log.level":"warn","@timestamp":"2024-11-18T14:35:33.565-0300","log.logger":"reader_multiline","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/reader/multiline.(*messageBuffer).finalize","file.name":"multiline/message_buffer.go","file.line":125},"message":"Multiline message is too large, truncated to the limit of 500 lines or 10485760 bytes","service.name":"filebeat","ecs.version":"1.6.0"}

mergify[bot] commented 1 week ago

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @mauri870? 🙏. For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify[bot] commented 1 week ago

backport-8.x has been added to help with the transition to the new branch 8.x. If you don't need it please use backport-skip label and remove the backport-8.x label.

elasticmachine commented 1 week ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elastic / beats