search

fluent / fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
https://docs.fluentd.org/output/s3
314 stars 215 forks source link

Append file support #433

Open raultang opened 9 months ago

raultang commented 9 months ago

Is your feature request related to a problem? Please describe.

From my current understanding, log file is visible after a timekey interval or a buffer size. If we configure with short time range there should have many files generated in case log content itself is not big. If we configure with large time range then we need wait quite some time for log visibility.

Describe the solution you'd like

S3 support append semantic, so maybe we can implement with append file instead of put semantic.

Describe alternatives you've considered

File/WebHdfs plugin seems support append files.

Additional context

No response

daipom commented 9 months ago

Thanks for your request. I don't know why the current out_s3 plugin doesn't support the append feature. If anyone knows any reason, please let me know.

You mean that if we want to make a large file unit, we have to keep it in the buffer for a long period of time, and it is inconvenient, right?

If you have any sample implementations, I would appreciate it if you could share them. Or PRs are welcome!

raultang commented 9 months ago

You mean that if we want to make a large file unit, we have to keep it in the buffer for a long period of time, and it is inconvenient, right?

We want to make the log file visible as soon as possible: if with short timekey, there might have many small files.

If anyone knows any reason, please let me know.

Ok, After read s3 documents, S3 itself not support append :)

daipom commented 7 months ago

Sorry for my late response.

We want to make the log file visible as soon as possible: if with short timekey, there might have many small files.

I see!!

Ok, After read s3 documents, S3 itself not support append :)

Thanks! So, S3 support append semantic was not correct? Do you mean S3 does not support append, so it seems difficult to add append feature to out_s3?