single_value format not taking affect for s3 plugin

[tgilman]

Describe the bug

I'm attempting to use the single_value format with the s3 plugin and it doesn't appear to be taking affect. Here's my Output configuration:

<match his.*>
  @type copy

  <store>
  @type forward
  time_as_integer true
  send_timeout 60s
  recover_wait 10s
  hard_timeout 60s

  <server>
    name logstash1
    host localhost
    port 5044
    weight 60
  </server>
  </store>
  <store>
    @type s3
    <format>
        @type single_value
        message_key log
    </format>
    s3_bucket fluend-poc
    s3_region us-west-2
    <buffer tag,time>
    @type file
    path /var/log/fluent/s3
    timekey 60 # 1 hour partition
    timekey_wait 1m
    timekey_use_utc true # use utc
    chunk_limit_size 256m
  </buffer>
  </store>
  <store>
  @type stdout
  </store>
</match>

The result still contains the full log entry:

2022-02-25T08:13:58+00:00   his.69dev   {"log":"2022-02-25_08:13:58.021 UTC DEBUG IncontactHubProcessor  - Token Refresh Cycle"}

The goal is to "reconstitute" the original log within the S3 bucket without the fluent metadata.

To Reproduce

See the above configuration.

Expected behavior

The result should be:

2022-02-25_08:13:58.021 UTC DEBUG IncontactHubProcessor  - Token Refresh Cycle

Your Environment

- Fluentd version: 1.14.4
- TD Agent version: Calyptia Windows Package
- fluent-plugin-s3 version: 1.6.1
- aws-sdk-s3 version:
- aws-sdk-sqs version:
- Operating system: Widnows Server 2019
- Kernel version: -

Your Configuration

See configuration above.

Your Error Log

2022-02-25 22:05:08 +0000 [info]: starting fluentd-1.14.4 pid=3136 ruby="3.0.3"
2022-02-25 22:05:08 +0000 [info]: spawn command to main:  cmdline=["C:/opt/calyptia-fluentd/bin/ruby.exe", "-Eascii-8bit:ascii-8bit", "C:/opt/calyptia-fluentd/bin/fluentd", "-c", "C:\\opt\\calyptia-fluentd\\etc\\calyptia-fluentd\\calyptia-fluentd.conf", "-o", "C:\\opt\\calyptia-fluentd\\calyptia-fluentd.log", "-x", "fluentdwinsvc", "--under-supervisor"]
2022-02-25 22:05:10 +0000 [info]: adding match pattern="debug.**" type="stdout"
2022-02-25 22:05:11 +0000 [info]: adding match pattern="his.*" type="copy"
2022-02-25 22:05:11 +0000 [info]: #0 adding forwarding server 'logstash1' host="localhost" port=5044 weight=60 plugin_id="object:b2c"
2022-02-25 22:05:13 +0000 [warn]: #0 The default value of s3_object_key_format will use ${chunk_id} instead of %{index} to avoid object conflict in v2
2022-02-25 22:05:13 +0000 [info]: adding source type="forward"
2022-02-25 22:05:13 +0000 [info]: adding source type="http"
2022-02-25 22:05:13 +0000 [info]: adding source type="debug_agent"
2022-02-25 22:05:13 +0000 [info]: #0 starting fluentd worker pid=2692 ppid=3136 worker=0
2022-02-25 22:05:13 +0000 [info]: #0 [input_debug_agent] listening dRuby uri="druby://127.0.0.1:24230" object="Fluent::Engine" worker=0
2022-02-25 22:05:13 +0000 [info]: #0 [input_forward] listening port port=24284 bind="0.0.0.0"
2022-02-25 22:05:13 +0000 [info]: #0 fluentd worker is now running worker=0
2022-02-25 22:05:01.707000000 +0000 his.69dev: {"log":"2022-02-25_22:05:01.707 UTC DEBUG IncontactHubProcessor  - Token Refresh Cycle"}

Additional context

No response

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days