logstash-plugins / logstash-input-s3

Apache License 2.0
57 stars 150 forks source link

Add decorate(event) in case of stateful codecs like multiline #173

Closed duaraghav8 closed 2 years ago

duaraghav8 commented 5 years ago

The decorate() method wasn't being called in case of stateful codecs before en-queuing the event. This resulted in type, tags and add_field options being ineffective when a user uses them with this plugin. This PR fixes it

See issue #153

Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/

zuifanzhuce commented 5 years ago

I'm using s3 and multiline, adding "decorate(event)" into s3.rb doesn't fix my issue. However, after adding all those below, the issue is gone.

@codec.flush do |event| decorate(event)

  event.set("cloudfront_version", metadata[:cloudfront_version]) unless metadata[:cloudfront_version].nil?
  event.set("cloudfront_fields", metadata[:cloudfront_fields]) unless metadata[:cloudfront_fields].nil?

  if @include_object_properties
    event.set("[@metadata][s3]", object.data.to_h)
  else
    event.set("[@metadata][s3]", {})
  end

  event.set("[@metadata][s3][key]", object.key)

  queue << event
end
roeezab commented 4 years ago

I'm using s3 and multiline, adding "decorate(event)" into s3.rb doesn't fix my issue. However, after adding all those below, the issue is gone.

@codec.flush do |event| decorate(event)

  event.set("cloudfront_version", metadata[:cloudfront_version]) unless metadata[:cloudfront_version].nil?
  event.set("cloudfront_fields", metadata[:cloudfront_fields]) unless metadata[:cloudfront_fields].nil?

  if @include_object_properties
    event.set("[@metadata][s3]", object.data.to_h)
  else
    event.set("[@metadata][s3]", {})
  end

  event.set("[@metadata][s3][key]", object.key)

  queue << event
end

It worked for me as well. Thanks @duaraghav8 and @zuifanzhuce!

kaisecheng commented 2 years ago

The issue is fixed in 3.8.3