search

fluent / fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
https://docs.fluentd.org/output/s3
314 stars 218 forks source link

Bucket name from placeholder #363

Open rmontenegroo opened 3 years ago

rmontenegroo commented 3 years ago

This patch allows the use of placeholders to setup bucket names.

There is a new parameter s3_bucket_fallback to use in cases when at least one placeholder is not found amoung chunk keys.

rmontenegroo commented 3 years ago

@cosmo0920 why get_placeholders_time does not return any time placeholder if I set @s3_bucket to "test-%Y" for example? If I set @s3_bucket = "test-%Y-%m-%d" get_placeholders_time return only %d.

rmontenegroo commented 3 years ago

The thing is PlaceholderValidator does not allow a timekey larger than a day (https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/output.rb#L686) so it won't be possible to validate time placeholders for year and month using the validator. Am I right? Something like "test-%Y-%m" won't be accepted by validator as placeholders.

rmontenegroo commented 3 years ago

@cosmo0920 I implemented a simple ext_get_placeholders_time only to check if there are time markers (%Y %m %d %H %M %S) in a string. With this I am able to check time placeholders, and with Validator class I can check tag and keys placeholders. I hope it is satisfying.

cosmo0920 commented 3 years ago

The thing is PlaceholderValidator does not allow a timekey larger than a day (https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/output.rb#L686) so it won't be possible to validate time placeholders for year and month using the validator. Am I right?

Right.

github-actions[bot] commented 3 years ago

This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days

github-actions[bot] commented 3 years ago

This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days

github-actions[bot] commented 2 years ago

This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days