fluent / fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
https://docs.fluentd.org/output/s3
314 stars 218 forks source link

Getting shutdown after reading 1-2 records #319

Closed rverma-jm closed 4 years ago

rverma-jm commented 4 years ago

For some strange reason, the plugin keep getting shutdown after reading 1-2 events.

2020-04-30 04:50:36 +0000 [info]: using configuration file: <ROOT>
  <worker 0>
    <source>
      @type s3
      @id cloudtrail
      @label @cloudtrail
      @log_level "debug"
      s3_bucket "jm-audit-logs"
      s3_region "ap-south-1"
      add_object_metadata false
      <web_identity_credentials>
        role_arn "arn:aws:iam::xx:role/jm-audit-fluent"
        role_session_name "jm-audit-s3-event-audit-cloudtrail"
        web_identity_token_file "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
      </web_identity_credentials>
      <sqs>
        queue_name "jm-audit-s3-event-audit-cloudtrail"
      </sqs>
    </source>
    <label @cloudtrail>
      <match input.s3>
        @type stdout
        @id firehose_ok
      </match>
    </label>
  </worker>
  <system>
    workers 1
    root_dir "/var/log/fluentd"
  </system>
</ROOT>

Fluentd logs

2020-04-30 04:50:37 +0000 [info]: #0 adding match in @cloudtrail pattern="input.s3" type="stdout"
2020-04-30 04:50:37 +0000 [info]: #0 adding source type="s3"
2020-04-30 04:50:37 +0000 [info]: #0 starting fluentd worker pid=16 ppid=6 worker=0
2020-04-30 04:50:37 +0000 [debug]: #0 [cloudtrail] Succeeded to create S3 client
2020-04-30 04:50:38 +0000 [debug]: #0 [cloudtrail] Succeeded to verify API keys
2020-04-30 04:50:38 +0000 [debug]: #0 [cloudtrail] Succeeded to create SQS client
2020-04-30 04:50:38 +0000 [debug]: #0 [cloudtrail] Succeeded to get SQS queue URL
2020-04-30 04:50:38 +0000 [info]: #0 fluentd worker is now running worker=0

Reading cloudtrail logs, this is a big json and nested json

2020-04-30 04:50:38 +0000 [debug]: #0 [cloudtrail]  Records=[{"eventVersion"=>"2.1", "eventSource"=>"aws:s3", "awsRegion"=>"ap-south-1", "eventTime"=>"2020-04-30T04:50:22.568Z", "eventName"=>"ObjectCreated:Put", "userIdentity"=>{"principalId"=> .... }

2020-04-30 04:51:06 +0000 [info]: Received graceful stop
2020-04-30 04:51:07 +0000 [info]: #0 fluentd worker is now stopping worker=0
2020-04-30 04:51:07 +0000 [info]: #0 shutting down fluentd worker worker=0
2020-04-30 04:51:07 +0000 [info]: #0 shutting down input plugin type=:s3 plugin_id="cloudtrail"
2020-04-30 04:51:07 +0000 [info]: #0 shutting down output plugin type=:stdout plugin_id="firehose_ok"
2020-04-30 04:51:08 +0000 [warn]: #0 [cloudtrail] killing existing thread thread=#<Thread:0x00007f55afd92a40@in_s3@/usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.10.2/lib/fluent/plugin_helper/thread.rb:70 sleep>
2020-04-30 04:51:08 +0000 [warn]: #0 [cloudtrail] thread doesn't exit correctly (killed or other reason) plugin=Fluent::Plugin::S3Input title=:in_s3 thread=#<Thread:0x00007f55afd92a40@in_s3@/usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.10.2/lib/fluent/plugin_helper/thread.rb:70 aborting> error=nil

Couldn't figure out what is the issue, even in debug log level nothing is reflected.

cp38510 commented 3 years ago

@rverma-jm Hello!) how did you solve the error?