logstash-plugins / logstash-input-s3

Apache License 2.0
57 stars 150 forks source link

Add last modified time of file in S3 to @metadata #152

Open sbenderli opened 6 years ago

sbenderli commented 6 years ago

To clarify, last modified time refers to what S3 stores for each file, ie the same data that is written to sincedb. I'd like this information to be available in @metadata.

This information would be useful for cases where timestamp information is not included in the files being processed, but you want to track when these files were uploaded to S3. The user could then calculate information such as "how long did it take for this file to be processed since it was put in S3".

The last_modified time could be written to @metadata here, for example: https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L374 Similar to how [@metadata][s3][key] is updated.