Buffers piling up

[atombender]

I'm seeing the GCS plugin uploading to the bucket. But it also leaves behind a lot of buffer files, up to the point where the queue becomes full. What am I doing wrong?

I deleted the buffers last night, and as you can see, it's building up constantly:

My config:

<match **>
  @type copy
  <store>
    @type file

    path /fluentd/log/everything
    append true
    buffer_type memory
    flush_interval 5s
    format json
    include_time_key
    time_format %Y-%m-%dT%H:%M:%S.%N
    time_slice_format %Y-%m-%d-%H
    disable_retry_limit
    utc
  </store>
  <store>
    @type gcs

    format json
    include_time_key
    disable_retry_limit
    utc
    buffer_type file
    buffer_path /fluentd/log/gcs.buffer
    buffer_chunk_limit 10M
    # Max 1000 * 10M = 10GB
    buffer_queue_limit 1000
    time_slice_format %Y%m%d%H
    time_slice_wait 10m

    project projectname
    bucket bucketname
    store_as gzip
    path fluentd/
    auto_create_bucket false
    keyfile "/etc/td-agent/secrets/gcs-service-account-key"
    storage_class nearline
  </store>
</match>

Lists of buffers and GCS data here.

Fluentd 0.12.35, plugin version 0.3.0.

[daichirata]

Sorry for the delay in replying. I was so busy with my work that I couldn't check the plug-ins.

It is probably not possible to write to GCS because of an authentication problem, but was there any error?

You may have already solved the problem, but we have released a new version that includes a response to fluent v1, so we recommend you try that.

[atombender]

This is such a long time ago that I don't remember any details. I just know that there were no errors in any logs.