search

y-ken / fluent-plugin-twitter

Fluentd Input/Output plugin to process tweets with Twitter Streaming API.
https://rubygems.org/gems/fluent-plugin-twitter
Other
52 stars 27 forks source link

Gem conflict error: json-2.0.2 conflicts with json (~> 1.8) #36

Open toddysm opened 7 years ago

toddysm commented 7 years ago

I am trying to set this up on Docker using 'fluentd-0.12.31' Fluentd image and it seems the plugin requires implicitly json-1.8 while fluent requires json-2.0. Is there any hard requirement for json-1.8 and can we update the dependencies?

2017-02-28 18:39:01 +0000 [info]: adding source type="twitter"
/usr/lib/ruby/2.3.0/rubygems/specification.rb:2284:in `raise_if_conflicts': Unable to activate twitter-5.17.0, because json-2.0.2 conflicts with json (~> 1.8) (Gem::ConflictError)
        from /usr/lib/ruby/2.3.0/rubygems/specification.rb:1407:in `activate'
        from /usr/lib/ruby/2.3.0/rubygems.rb:196:in `rescue in try_activate'
        from /usr/lib/ruby/2.3.0/rubygems.rb:193:in `try_activate'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:125:in `rescue in require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in `require'
        from /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-twitter-0.6.0/lib/fluent/plugin/in_twitter.rb:1:in `<top (r
equired)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:172:in `block in try_load_plugin'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:170:in `each'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:170:in `try_load_plugin'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:130:in `new_impl'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:55:in `new_input'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:149:in `add_source'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:95:in `block in configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:92:in `each'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:92:in `configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/engine.rb:129:in `configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/engine.rb:103:in `run_configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:489:in `run_configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:160:in `block in start'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:366:in `main_process'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:339:in `block in supervise'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:338:in `fork'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:338:in `supervise'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:156:in `start'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/bin/fluentd:5:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
y-ken commented 7 years ago

@toddysm

Hi. I have checked the json version in my td-agent2 latest environment which has installed fluent-plugin-twitter. It seems none requirement of json-2.0 by fluentd-0.12.31. Have you installed any other plugins? Let's show me the gem list.

$ td-agent-gem list

*** LOCAL GEMS ***

actionmailer (4.2.6)
actionpack (4.2.6)
actionview (4.2.6)
activejob (4.2.6)
activemodel (4.2.6)
activerecord (4.2.6)
activesupport (4.2.6)
addressable (2.5.0, 2.4.0)
arel (6.0.3)
aws-sdk (2.6.42)
aws-sdk-core (2.6.42)
aws-sdk-resources (2.6.42)
aws-sdk-v1 (1.66.0)
aws-sigv4 (1.0.0)
bigdecimal (1.2.4)
bson (1.12.5)
bson_ext (1.12.5)
buftok (0.2.0)
builder (3.2.2)
bundler (1.10.6)
bzip2-ffi (1.0.0)
celluloid (0.15.2)
cool.io (1.4.5)
diff-lcs (1.2.5)
domain_name (0.5.20170223)
draper (1.4.0)
equalizer (0.0.10)
erubis (2.7.0)
faraday (0.9.2)
ffi (1.9.14)
fluent-logger (0.6.1)
fluent-mixin-config-placeholders (0.4.0)
fluent-mixin-plaintextformatter (0.2.6)
fluent-plugin-kafka (0.4.1)
fluent-plugin-mongo (0.7.16)
fluent-plugin-rewrite-tag-filter (1.5.5)
fluent-plugin-s3 (0.8.0)
fluent-plugin-scribe (0.10.14)
fluent-plugin-td (0.10.29)
fluent-plugin-td-monitoring (0.2.2)
fluent-plugin-twitter (0.6.0)
fluent-plugin-webhdfs (0.4.2)
fluentd (0.12.31)
fluentd-ui (0.4.3)
font-awesome-rails (4.6.3.0)
globalid (0.3.7, 0.3.6)
haml (4.0.7)
haml-rails (0.5.3)
hike (1.2.3)
hirb (0.7.3)
http (1.0.4)
http-cookie (1.0.3)
http-form_data (1.0.1)
http_parser.rb (0.6.0)
httpclient (2.8.2.4)
i18n (0.7.0)
io-console (0.4.3)
ipaddress (0.8.3)
jbuilder (2.5.0)
jmespath (1.3.1)
jquery-rails (3.1.4)
json (1.8.1)
kramdown (1.11.0)
kramdown-haml (0.0.3)
loofah (2.0.3)
ltsv (0.1.0)
mail (2.6.4)
memoizable (0.4.2)
mime-types (3.1)
mime-types-data (3.2016.0521)
mini_portile2 (2.1.0)
minitest (5.9.0, 4.7.5)
mixlib-cli (1.7.0)
mixlib-config (2.2.4)
mixlib-log (1.7.1)
mixlib-shellout (2.2.7)
mongo (1.12.5)
msgpack (0.5.11)
multi_json (1.11.0)
multipart-post (2.0.0)
naught (1.1.0)
nokogiri (1.7.0)
ohai (6.20.0)
oj (2.18.0)
parallel (1.8.0)
psych (2.0.5)
public_suffix (2.0.4)
puma (3.4.0)
rack (1.6.4)
rack-test (0.6.3)
rails (4.2.6)
rails-deprecated_sanitizer (1.0.3)
rails-dom-testing (1.0.8)
rails-html-sanitizer (1.0.3)
railties (4.2.6)
rake (10.1.0)
rdoc (4.1.0)
request_store (1.1.0)
ruby-kafka (0.3.15)
ruby-progressbar (1.8.1)
rubyzip (1.2.0, 1.1.7)
sass (3.2.19)
sass-rails (4.0.5)
settingslogic (2.0.9)
sigdump (0.2.4)
simple_oauth (0.3.1)
sprockets (2.12.4)
sprockets-rails (2.2.4)
string-scrub (0.0.5)
sucker_punch (1.0.5)
systemu (2.5.2)
td (0.15.2)
td-client (0.8.85)
td-logger (0.3.26)
test-unit (2.1.10.0)
thor (0.19.1)
thread_safe (0.3.5)
thrift (0.8.0)
tilt (1.4.1)
timers (1.1.0)
twitter (5.17.0)
tzinfo (1.2.2)
tzinfo-data (1.2016.10)
unf (0.1.4)
unf_ext (0.0.7.2)
uuidtools (2.1.5)
webhdfs (0.8.0)
yajl-ruby (1.2.1)
zip-zip (0.3)
y-ken commented 7 years ago

@okkez Did you have ever faced any this related issue in other plugins?

okkez commented 7 years ago

No, I haven't. twitter gem(5.17.0) requires json gems ~>1.8.

Could you show me full configuration? @toddysm

y-ken commented 7 years ago

thanks @okkez

toddysm commented 7 years ago

Here is the full gem list that I have on my machine:

addressable (2.5.0) azure (0.7.7, 0.6.4) azure-core (0.1.6) bigdecimal (1.2.8) buftok (0.2.0) bundler (1.13.7) cool.io (1.4.5 x64-mingw32) did_you_mean (1.0.0) domain_name (0.5.20170223) equalizer (0.0.10) eventmachine (1.2.3 x64-mingw32) faraday (0.10.1, 0.9.2) faraday_middleware (0.10.1) ffi (1.9.14 x64-mingw32) ffi-win32-extensions (1.0.3) fluent-mixin-config-placeholders (0.4.0) fluent-plugin-azurestorage (0.0.8) fluent-plugin-twitter (0.6.0) fluentd (0.14.11 x64-mingw32, 0.12.31) http (1.0.4) http-cookie (1.0.3) http-form_data (1.0.1) http_parser.rb (0.6.0) io-console (0.4.5) json (2.0.3, 1.8.5, 1.8.3) macaddr (1.7.1) memoizable (0.4.2) mime-types (3.1, 1.25.1) mime-types-data (3.2016.0521) mini_portile2 (2.1.0) minitest (5.8.5) msgpack (1.0.2 x64-mingw32, 0.5.12) multipart-post (2.0.0) naught (1.1.0) net-telnet (0.1.1) nokogiri (1.7.0.1 x64-mingw32) power_assert (0.4.1, 0.2.6) psych (2.1.0) public_suffix (2.0.5) rake (12.0.0, 10.4.2) rdoc (4.2.1) rr (1.2.0) serverengine (2.0.4 x64-mingw32) sigdump (0.2.4) simple_oauth (0.3.1) string-scrub (0.0.5) strptime (0.1.9 x64-mingw32) systemu (2.6.5) test-unit (3.2.3, 3.1.5) test-unit-rr (1.0.5) thor (0.19.4) thread_safe (0.3.5) twitter (5.17.0) tzinfo (1.2.2) tzinfo-data (1.2016.10) unf (0.1.4) unf_ext (0.0.7.2 x64-mingw32) uuid (2.3.8) uuidtools (2.1.5) win32-api (1.6.0 universal-mingw32) win32-event (0.6.3) win32-ipc (0.6.6) win32-service (0.8.10) windows-api (0.4.4) windows-pr (1.2.5) yajl-ruby (1.3.0)

y-ken commented 7 years ago

I think that you have installed fluentd into your existing environment. Some other gems have require to json-2.0.3 which are not comes from fluentd.

Would you please try to use independent ruby environment for fluentd? http://docs.fluentd.org/v0.14/articles/install-by-msi

toddysm commented 7 years ago

After installing into an independent ruby environment I get the following error:

Gem::ConflictError error="Unable to activate twitter-5.17.0, because faraday-0.10.1 conflicts with faraday (~> 0.9.0)"

I guess the reason is that you guys are using fluentd (0.12.31) while I am using fluentd (0.14.11)

toddysm commented 7 years ago

@okkez and @y-ken I think the problem is that twitter gem(5.17.0) requires json gems ~>1.8. Is it possible to migrate to latest twitter gem 6.1, which works well with json ~>2.0?

ottimo commented 7 years ago

I had the same problem with fluentd docker image: when you build the image, dockerfile runs the command "gem install json" and this will install last version of the gem (~ > 2.0). So I found a workaround: I use a modified version of fluentd dockerfile where force the installation of version 1.8 of json and it seems to work. (gem install json -v "~> 1.8"). I hope it's useful.

y-ken commented 7 years ago

@okkez I'm afraid could you please help me the migration to twitter 6.x?

okkez commented 7 years ago

OK. I will work for this in early April.

y-ken commented 7 years ago

Thank you very much.

y-ken commented 7 years ago

status has updated. ref #39

y-ken commented 7 years ago

Hi there (cc: @toddysm @ottimo ) Today, I have released new fluent-plugin-twitter gem. https://rubygems.org/gems/fluent-plugin-twitter/versions/0.6.1

-  s.add_runtime_dependency "twitter", "~> 5.0"
+  s.add_runtime_dependency "twitter", "~> 6.0"

Would you please try it on?