How to set timezone in fluent bit S3 plugin?

[JalanJiang]

As the title.

[PettitWesley]

@JalanJiang I think Time_Offset in the parser is possibly part of what you want: https://docs.fluentbit.io/manual/pipeline/parsers/configuring-parser

Basically, when you ingest log records, their timestamps can be obtained via parsing the log.

-> What is the exact question here? Is it about the timestamp used in the S3 key format or about the timestamp of the actual log records?

However, specifically for S3, unfortunately the timestamp is currently not based on the log records at all: https://github.com/aws/aws-for-fluent-bit/issues/459

Also, S3 key uses gmtime_r, which is UTC: https://pubs.opengroup.org/onlinepubs/009696799/functions/gmtime.html

Also, I want to note that there is a report Time_Offet is broken on windows: https://github.com/fluent/fluent-bit/issues/6121

[PettitWesley]

Time zones are a human construct that change constantly based on changing laws and etc. So we would probably support a Time_Offset for S3 same as is seen in the Parser above. Which allows you to say your timestamps are fixed offset from UTC.

[PettitWesley]

@DrewZhang13 Made us realize that this could be a feature used in other plugins like kinesis_firehose and kinesis_streams, which has a time_key_format and thus could also have a time_offset so that your time_key is in your local timezone.

We should also look into seeing if there is a way to discover the local time offset or not. But probably it has to be a user config.

[jasminehung]

Need this feature! Would be great if the timezone in the file name could be modified s3_key_format /${POD_NAME}-%Y-%m-%d-%H-%M.log

[maiconbaum]

Any news?