cosmo0920 / fluent-bit-go-s3

[Deprecated] The predessor of fluent-bit output plugin for Amazon S3. https://aws.amazon.com/s3/
Apache License 2.0
33 stars 13 forks source link

How to control the upload file's size in S3 #39

Open grontorino opened 4 years ago

grontorino commented 4 years ago

I'm confused how to set the Chunk_size to control the upload file size, it's seems not work after modify INPUT or SERVICE.
No matter how I modify the Buffer_Size or Buffer_Chunk_Size, the uploaded file size is still about 540Kb. Is there any other way?

2020-06-19 07:42:53     546653 20200619074252.log.gz
2020-06-19 07:42:55     547130 20200619074254.log.gz
2020-06-19 07:42:56     547566 20200619074255.log.gz
2020-06-19 07:42:57     548861 20200619074256.log.gz
2020-06-19 07:42:58     282101 20200619074257.log.gz

Here is my config:

[SERVICE]
    Flush        300
    Daemon       Off
    Log_Level    warning
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

    #log_Level                 info
    #storage.path              /var/log/
    #storage.sync              normal
    #storage.checksum          off
    #storage.backlog.mem_limit 200M

[INPUT]
    Name forward
    Host 0.0.0.0
    #storage.type  filesystem
    Buffer_Size 200M
    Buffer_Chunk_Size  10M
cosmo0920 commented 4 years ago

For golang plugin, there is no way to control emitting size. Could you send your feature request to fluent-bit-go repository?