Proposal to support TimeFormat

k-kinzal commented 4 years ago

Problem

The current path to save to S3 is /[path prefix]/YYYYMMDD/HH. However, I wanted to save in Hive format, like/[path prefix]/dt=YYYY-MM-DD, in order to use AWS Athena.

So can we add an option like TimeFormat to format the date part of the path?

[Output]
    Name s3
    Match *
    AccessKeyID     yourawsaccesskeyid
    SecretAccessKey yourawssecretaccesskey
    Bucket          yourbucketname
    S3Prefix yours3prefixname
    TimeFormat dt=%Y-%m-%d
    Region us-east-1
    Compress gzip

The value of TimeFormat might be better to use Golang's format, such as TimeFormat dt=2006-01-02.

Steps to replicate

None. This is feature request.

Expected Behavior or What you need to ask

...

Using Fluentd and S3 plugin versions

OS version
Bare Metal or within Docker or Kubernetes or others?
fluent-bit version
- paste result of fluent-bit --version or td-agent-bin --version
fluent-bit-go-s3 version
- paste boot log of fluent-bit or td-agent-bit

k-kinzal commented 4 years ago

I implemented it in Go's time format for reference. https://github.com/k-kinzal/fluent-bit-go-s3/commit/28aa3d3e25df883a2e6726d11b2c9b8bca1493cd This is the easiest implementation, but I doubt it's user-friendly.

cosmo0920 commented 4 years ago

Hmm, it looks reasonable for me!

k-kinzal commented 4 years ago

@cosmo0920 Thanks for taking a look at my proposal! So I'll send you a PR in Go's time format.

cosmo0920 / fluent-bit-go-s3