Closed umarengasamy closed 2 years ago
I had a similar issue. Did you do any filtering on the s3 key that's usually part of every event's metadata? Looking at the plugin's code, I think that the last line is actually flushed to the pipeline, but it's not initialized with the s3 metadata which might cause unexpected failures of filters. At least that's what happened to me. See #190 for the changes I made to solve my issue.
I use s3 key , mutate and make it as fields. We have issue only when we use multiline codec.
We are experiencing a similar strange behavior
%{[@metadata][s3][key]} is assigned to a field PATH, but during processing of the multiline data at some seemingly random points the metadata is lost and PATH becomes the string "%{[@metadata][s3][key]}"
input {
s3 {
access_key_id => ""
secret_access_key => ""
bucket => ""
region => ""
prefix => ""
exclude_pattern => "^((?!\.xml).)*$"
codec => multiline {
charset => "ISO8859-1"
pattern => "<DataSection>"
negate => true
what => "previous"
}
additional_settings => {
force_path_style => true
follow_redirects => false
}
}
}
filter {
# ADD PATH field
mutate {
add_field => {
"PATH" => "%{[@metadata][s3][key]}"
}
}
This seems to be because the input plugin does not decorate events when flushing stateful codecs to the queue https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L242
Is this issue a dupe of #153 ?
The issue is fixed in 3.8.3
Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.
For all general issues, please provide the following details for fast resolution:
Steps to Reproduce: input { s3 { type => "multilinelogs" bucket => "" region => "" interval => "1" prefix => "xxx" delete => true role_session_name => "" sincedb_path => "/dev/null" additional_settings => { force_path_style => true follow_redirects => false } codec => multiline { pattern => "^\s" what => "previous" auto_flush_interval => 1 } }
}