FluentD S3 output no name in directory

[vaibhavt19]

FluentD Version: fluentd:v1.6 Issue: FluentD writing the data on two places one directory name is blank FluentD configuration.

  <store>
    @type s3
    s3_bucket log-analytics
    s3_region us-east-1
    path telemetry/phoenix/${type}/%Y/%m/%d/%H/
    s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
    <format>
      @type json
    </format>
    store_as json
    <buffer time,type>
      @type memory
      timekey 3600 # 1 hour partition
      timekey_wait 10m
      timekey_use_utc true # use utc
      chunk_limit_size 256m
      time_slice_format %Y-%m-%d/%H
      flush_interval 15s
    </buffer>
  </store>

S3 Output

Logs are visible on both locations.

[repeatedly]

Maybe some records doesn't have type field in the record. Does the log in empty name directory have the expected fields?

[vaibhavt19]

Yes, you are right. But these logs are not generated by us. Somehow FluentD generating its own logs and parsing it to output. No idea where to configure this or it's the default behavior.

{"Write":null,"chunk":null,"58edc4dd85279c7671b2ef578b83ef59":null,"/":null,"27":null,"records":null,"5":null,"KB":null}
{"Finish":null,"writing":null,"chunk":null}
{"Write":null,"chunk":null,"58edc516839930fd2fceac0ec69039d4":null,"/":null,"31":null,"records":null,"6":null,"KB":null}
{"Finish":null,"writing":null,"chunk":null}
{"Write":null,"chunk":null,"58edc54fdb97c2fb0d92e6c1d03b39c6":null,"/":null,"19":null,"records":null,"3":null,"KB":null}
{"Finish":null,"writing":null,"chunk":null}
{"Write":null,"chunk":null,"58edc5891fead38e9cd3d2952e2c5d69":null,"/":null,"17":null,"records":null,"3":null,"KB":null}
{"Finish":null,"writing":null,"chunk":null}

[repeatedly]

Hmm... fluentd itself doesn't generate such logs. Here is the result of grep with Finish.

% grep -rne 'Finish' lib/fluent/
lib/fluent//plugin_helper/http_server/server.rb:60:          @logger.debug('Finished HTTP server')

So maybe, I assume 3rd party plugins generate it or something.

[vaibhavt19]

Thanks for the guidance. I have disabled the logs from fluentd as well on Docker containers. But still below logs getting generated on the S3 output. Could it be an issue with the docker itself?

{"fluentd":null,"worker":"0","is":null,"now":null,"stopping":null}
{"fluentd":null,"worker":"0","is":null,"now":null,"running":null}

Docker compose
version: '2'
services:
  fluentd:
    build: ./fluentd
    network_mode: bridge
    hostname: fluentd
    container_name: fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
      - ./firehose/:/firehose
      - /var/log/:/log/
    ports:
      - "1517:1517/udp"
      - "1517:1517/tcp"
      - "1518:1518/tcp"
      - "1518:1518/udp"
    logging:
        driver: none

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[kenhys]

It seems that this is not plugin issue https://github.com/fluent/fluent-plugin-s3/issues/282#issuecomment-516686063