/var/log/docker buffering to disk?

[Strainy]

I'm running my containers in AWS ECS on the ECS optimised AMI.

I've set the following docker configuration:

                - content: |
                    DAEMON_MAXFILES=1048576
                    DAEMON_PIDFILE_TIMEOUT=10
                    OPTIONS="$OPTIONS --log-driver fluentd --log-opt fluentd-address=localhost:24224 --log-opt tag="docker.{{.ImageName}}" --log-opt labels=${LogLabels} --log-opt fluentd-async-connect=true"
                  path: /etc/sysconfig/docker

My volumes are now filling up due to a file called /var/log/docker which appears to be growing unconstrained.

I'm using the following FluentD configuration (I'm also running FluentD in a docker container):

<source>
  @type  forward
  @id    input1
  @label @mainstream
  port  24224
</source>

<filter docker.**>
  @type stdout
</filter>

<label @mainstream>
  <match docker.**>
    @type kinesis_streams
    region "#{ENV['REGION']}"
    stream_name "#{ENV['STREAM_NAME']}"
    num_threads 4

    # Limit Memory Buffer
    buffer_type memory
    buffer_path /fluentd/log/fluent.matcher*
    buffer_chunk_limit "#{ENV['BUFFER_CHUNK_LIMIT']}"
    buffer_queue_limit "#{ENV['BUFFER_QUEUE_LIMIT']}"
    buffer_queue_full_action block
    max_retry_wait 30
    flush_interval "#{ENV['FLUSH_INTERVAL']}"

  </match>
</label>

I've attempted to configure logrotate, but have noticed some strange behaviour where files with the current date appended multiple times are created until (I assume) logrotate doesn't recognise the file anymore and then that file fills up the disk.

I'm guessing the issue is due to FluentD not being able to ship the logs to Kinesis fast enough. So the FluentD docker plugin must be creating a secondary buffer on the file system.

Does anyone have any suggestions for preventing this issue?

[riywo]

Thank you for reporting, but I have several questions:

What types of events are stored in /var/log/docker?

Do you use fluentd logging driver even with the fluentd container? I wonder that the filter plugin uses stdout, so it sends the event to container's stdout, then logging driver forward it to the container again, and continue.

Why do you think the fluentd can't send logs to Kinesis so fast?

[Strainy]

Hi @riywo,

The events stored look to be from the Docker daemon itself. The file contains the log line that Docker was unable to forward to fluentd with an error message. Something like:

Aug 7 17:45:11 ip-10-0-60-181 docker[65207]: time="2017-08-07T17:45:11.863947825-07:00" level=error msg="Failed to log msg \"<FULL TEXT OF THE MESSAGE>\" for logger fluentd: fluent#send: can't send logs, client is reconnecting"

The fluentd logging driver is used with the fluentd container as it'll just use the default logging driver set.

Looking at the documentation, the stdout filter plugin does look to print everything to stdout! So it could just be the logs are getting looped haha.

Still, the problem remains. It appears the overflow from this high load results in the daemon or fluentd driver itself logging these failures out to disk and the file is not being rotated.

[riywo]

The message seems to be generated by fluentd logging driver. So, you should first check why the reconnection happened and it might not be caused by this plugin directly. You can ask to docker or fluentd community, but this is not the place. I'm closing this issue.

[Strainy]

For those experiencing this issue, I've raised an issue on the AWS-ECS-Init repo. They should be able to point me in the right direction: https://github.com/moby/moby/issues/34974

[riywo]

Yes, that is the right place! Thank you for keeping update here.

I'm just pointing the ecs-init issue for reference: https://github.com/aws/amazon-ecs-init/issues/119