search

fluent / fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
https://docs.fluentd.org/output/s3
314 stars 218 forks source link

S3 Input: Need S3 Meta-information like Path in bucken #277

Closed kochp closed 5 years ago

kochp commented 5 years ago

Hi everybody,

I am using the Plugin as imput from an S3 bucket. Therefore the files will be stored in a specific path blow the S3 bucket. This Information is important for me to know.

Is there a possibility to add this meta information to the Log-Message or as part of the tag?

I did not find any option in the direction or did I miss something?

atkinsj commented 5 years ago

+1. Having the object key exposed as a variable to be used in routing would be helpful.

The specific use case I have in mind is where I place multiple different inputs in one s3 bucket behind one input.s3 and then use the object key prefix to route them differently. I use rewrite_tag_filter to route them differently based on a message regex which is a bigger performance penalty than I'd like to wear.

kochp commented 5 years ago

I just found this pull request (https://github.com/fluent/fluent-plugin-s3/pull/220) and tried with an locally created gem based on that PR and it worked fine.

I am just wondering why it is not merged into the this project.

ganmacs commented 5 years ago

FYI: https://github.com/fluent/fluent-plugin-s3/releases/tag/v1.1.11 released

https://github.com/fluent/fluent-plugin-s3/blob/15915d83e9461929e19a78fa0d5a01fbcf6ceb39/ChangeLog#L1-L4