search

ixixi / fluent-plugin-sqs

Store fluent-event as queue message to Amazon SQS.
35 stars 53 forks source link

fluent-plugin-sqs-2.1.0 breaks td-agent-2.3.6 #34

Open dwoo4dwoo opened 6 years ago

dwoo4dwoo commented 6 years ago

Hello,

The latest fluent-plugin-sqs-2.1.0 breaks td-agent-2.3.5. See below. fluent-plugin-sqs-2.0.0 is fine.

I see commits by @cosmo0920 to support v0.14; however, the gemspec implies this should still work for v0.12.

David

% td-agent
2017-09-12 23:11:31 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2017-09-12 23:11:31 +0000 [info]: starting fluentd-0.12.36
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-kafka' version '0.5.5'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-mongo' version '0.8.0'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-s3' version '0.8.2'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-sqs' version '2.1.0'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-td' version '0.10.29'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.7.0'
2017-09-12 23:11:31 +0000 [info]: gem 'fluentd' version '0.12.36'
2017-09-12 23:11:31 +0000 [info]: adding match pattern="docker.**" type="sqs"
/opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- fluent/plugin/output (LoadError)
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-sqs-2.1.0/lib/fluent/plugin/out_sqs.rb:1:in `<top (required)>'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:172:in `block in try_load_plugin'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:170:in `each'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:170:in `try_load_plugin'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:130:in `new_impl'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:59:in `new_output'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/agent.rb:131:in `add_match'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/agent.rb:64:in `block in configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/agent.rb:57:in `each'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/agent.rb:57:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/root_agent.rb:86:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/engine.rb:129:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/engine.rb:103:in `run_configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:489:in `run_configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:160:in `block in start'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:366:in `call'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:366:in `main_process'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:339:in `block in supervise'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:338:in `fork'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:338:in `supervise'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:156:in `start'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/bin/fluentd:5:in `<top (required)>'
        from /opt/td-agent/embedded/bin/fluentd:23:in `load'
        from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
        from /sbin/td-agent:7:in `load'
        from /sbin/td-agent:7:in `<main>'
2017-09-12 23:11:31 +0000 [info]: process finished code=256
2017-09-12 23:11:31 +0000 [warn]: process died within 1 second. exit.
cosmo0920 commented 6 years ago

@dwoo4dwoo Could you send a PR to fix this issue? Or you want to help me to fix this issue?

dwoo4dwoo commented 6 years ago

@cosmo0920 - according to https://docs.fluentd.org/v0.14/articles/plugin-update-from-v12:

Plugins which uses v0.14 API don’t work on Fluentd v0.12.x.

So your changes to support v0.14 definitely break v0.12. There must be something wrong with the gemspec, since I can install fluent-plugin-sqs-2.1.0 with fluentd v0.12, and this combination will never work.

@ixixi - the same doc https://docs.fluentd.org/v0.14/articles/plugin-update-from-v12 recommends creating a fluentd-v0.12 branch so you're able to fix bugs and release features for v0.12. Or did you intentionally abandon v0.12?

Also, according to https://docs.fluentd.org/v0.12/articles/plugin-development#plugin-versioning-policy, the major version should have changed when moving to v0.14. It's not at all intuitive that 2.0.0 works with v0.12 whereas 2.1.0 breaks v0.12.

For now, I'm going to hardcode using fluent-plugin-sqs v2.0.0, so I get v2 of the AWS SDK.

dwoo4dwoo commented 6 years ago

It'd also be nice to have something akin to https://github.com/uken/fluent-plugin-elasticsearch#requirements in README.md.

cosmo0920 commented 6 years ago

Patch is welcomed.

dwoo4dwoo commented 6 years ago

Meh, I didn't break it, I'm not going to fix it :-)

cosmo0920 commented 6 years ago

But you've hit this issue and reported first. It is enough reason to fix it, though. :-P

dwoo4dwoo commented 6 years ago

For now, I'm going to hardcode using fluent-plugin-sqs v2.0.0, so I get v2 of the AWS SDK.

Sigh, I'm now hitting https://github.com/ixixi/fluent-plugin-sqs/issues/30, so I'm going back to v1.8.0 which I know works.

cosmo0920 commented 6 years ago

https://github.com/ixixi/fluent-plugin-sqs/pull/36 will fix this issue. Thank you for your report.

dwoo4dwoo commented 6 years ago

36 will fix this issue.

Awesome - thanks @cosmo0920 !