search

y-ken / fluent-plugin-twitter

Fluentd Input/Output plugin to process tweets with Twitter Streaming API.
https://rubygems.org/gems/fluent-plugin-twitter
Other
53 stars 27 forks source link

conflict http_parser.rb #5

Closed inokappa closed 10 years ago

inokappa commented 10 years ago

Dear y-ken san

I'm inokappa.

I tried td-agent and fluent-plugin-twitter. I got an error message(please confirm following "message and backtrace") while trying to "/etc/init.d/td-agent configtest".

message and backtrace

and other gems.

/usr/lib/fluent/ruby/bin/fluent-gem list | grep tw
em-twitter (0.3.1)
fluent-plugin-twitter (0.2.2)
tweetstream (2.6.0)
twitter (4.8.1)

td-agent.conf

<source>
  type twitter
  consumer_key        xxxxxxxxxxxxxxxxxxxxxx
  consumer_secret     yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
  oauth_token         zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz                      
  oauth_token_secret  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  tag                 input.twitter.sampling  
  timeline            sampling
  keyword             hoge
  lang                ja,en
</source>

<match input.twitter.sampling>
  type stdout
</match>

Any advice or information you could provide would be greatly appreciated.

Thank you.

y-ken commented 10 years ago

It's very sorry for inconvenience. For quick patch, it's supposed to be fixed with locking depended gem version like that url refered in comment. https://github.com/y-ken/fluent-plugin-twitter/issues/4#issuecomment-19982376

Could you please update gemspec file localy?

y-ken commented 10 years ago

I'll repro on my environment later.

inokappa commented 10 years ago

@y-ken

Thank you for your reply. I'll try update gemspec later.

Thank you.

y-ken commented 10 years ago

I have confirmed not woking correctly.

$ sudo /usr/lib64/fluent/ruby/bin/gem install fluent-plugin-twitter
ERROR:  While executing gem ... (Gem::DependencyError)
    Unable to resolve dependencies: em-twitter requires http_parser.rb (< 0.7, >= 0.6.0.beta.2); em-http-request requires http_parser.rb (>= 0.6.0.beta.2)
y-ken commented 10 years ago

it is a quick patch. I'll release it as v0.2.3 https://github.com/y-ken/fluent-plugin-twitter/commit/6a7d096

y-ken commented 10 years ago

Would you please check it out the version 0.2.3 of fluent-plugin-twitter which I released now? http://rubygems.org/gems/fluent-plugin-twitter

inokappa commented 10 years ago

@y-ken

Thank you for your reply. I'll try version 0.2.3 immediately.

Thank you.

inokappa commented 10 years ago

@y-ken

I tried version 0.2.3. Please confirm following.

environment

result

I execute following command.

/usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-twitter --no-ri --no-rdoc -V

I got error following.

ERROR:  While executing gem ... (Gem::DependencyError)
    Unable to resolve dependencies: fluent-plugin-twitter requires em-twitter (= 0.2.2); tweetstream requires em-twitter (~> 0.3); em-http-request requires http_parser.rb (>= 0.6.0.beta.2)

Any advice or information you could provide would be greatly appreciated.

Thank you.

inokappa commented 10 years ago

@y-ken

I using check with docker container. Please use a container if you'd like.

How to use.(In Host OS Ubuntu case)

  1. Please install docker(apt-get install lxc-docker)
  2. docker pull inokappa/td-agent_sandbox
  3. docker run -t -i inokappa/td-agent_sandbox /bin/bash

After booting normally try to install the "fluent-plugin-twitter". I appreciate your help.

Thank you.

（※今更ながらですが変な英語が随所に出てしまい本当に申し訳ございません。何卒、宜しくお願い致します）

y-ken commented 10 years ago

Hi, @inokappa san

I have confirmed the issue on my clean installed centos 6.4 environment that acts same behavior on ubuntu. It's very sorry for it.

$ sudo /usr/lib64/fluent/ruby/bin/gem install fluent-plugin-twitter
ERROR:  While executing gem ... (Gem::DependencyError)
    Unable to resolve dependencies: fluent-plugin-twitter requires em-twitter (= 0.2.2); tweetstream requires em-twitter (~> 0.3); em-http-request requires http_parser.rb (>= 0.6.0.beta.2)

And now, dependency fixed gem has works fine by the commit of https://github.com/y-ken/fluent-plugin-twitter/commit/d684470 I'll yank the 0.2.3 and release 0.2.4 soon.

# /usr/lib64/fluent/ruby/bin/gem list

*** LOCAL GEMS ***

addressable (2.3.5)
aws-sdk (1.8.3.1)
bigdecimal (1.1.0)
bson (1.8.6)
bson_ext (1.8.6)
bundler (1.2.5)
cookiejar (0.3.0)
cool.io (1.1.1)
daemons (1.1.9)
em-http-request (1.0.3)
em-socksify (0.3.0)
em-twitter (0.2.2)
eventmachine (1.0.3)
faraday (0.8.8)
fluent-logger (0.4.6)
fluent-mixin-config-placeholders (0.2.3)
fluent-mixin-plaintextformatter (0.2.2)
fluent-plugin-flume (0.1.1)
fluent-plugin-mongo (0.7.1)
fluent-plugin-s3 (0.3.4)
fluent-plugin-scribe (0.10.10)
fluent-plugin-td (0.10.16)
fluent-plugin-twitter (0.2.3)
fluent-plugin-webhdfs (0.2.1)
fluentd (0.10.39)
git (1.2.6)
hirb (0.7.1)
http_parser.rb (0.5.3, 0.5.1)
httpclient (2.3.4.1)
io-console (0.3)
iobuffer (1.1.2)
jeweler (1.6.2)
json (1.7.7, 1.5.4)
ltsv (0.1.0)
mini_portile (0.5.1)
minitest (2.5.1)
mongo (1.8.6)
msgpack (0.4.7)
multi_json (1.8.2)
multipart-post (1.2.0)
nokogiri (1.5.10)
parallel (0.6.5)
rake (0.9.2.2)
rdoc (3.9.4)
rubyzip (0.9.9)
simple_oauth (0.2.0)
td (0.10.89)
td-client (0.8.55)
td-logger (0.3.23)
thrift (0.8.0)
tweetstream (2.5.0)
twitter (4.8.1)
uuidtools (2.1.4)
webhdfs (0.5.3)
yajl-ruby (1.1.0)
inokappa commented 10 years ago

Hi, @y-ken san

Thank you for your reply. I am looking forward to version 0.2.4.

Thank you;-)

y-ken commented 10 years ago

Hi, @inokappa san

I have released fluent-plugin-twitter v0.2.4 now. Would you please check it out again? http://rubygems.org/gems/fluent-plugin-twitter

inokappa commented 10 years ago

@y-ken san

Thank you for your reply.

OK. I'll try later. Please just a moment.

Thank you!

inokappa commented 10 years ago

@y-ken san

I tried version 0.2.4. Plugin Install and td-agent configtest is no problem. gem conflict solved !!

But My environment does not start "td-agent". It's following td-agent.log

2013-12-03 10:34:10 +0000 [info]: process finished code=134
2013-12-03 10:34:10 +0000 [warn]: process died within 1 second. exit.

Since there is probably a problem in my environment, I would like to investigate.

Thank you.

y-ken commented 10 years ago

Hi, @inokappa san

I'm very sorry for my lacking QA test again and again. It seems fine but, it got died after a moment. I think it have to support 5.x version of this gem http://rubygems.org/gems/twitter

$ sudo  /etc/init.d/td-agent start
Starting td-agent:                                         [  OK  ]
2013-12-04 14:42:14 +0900 [info]: twitter: starting Twitter Streaming API for userstream. tag:debug.twitter.userstream lang:ja,en
2013-12-04 14:42:15 +0900 [info]: process finished code=134
2013-12-04 14:42:15 +0900 [warn]: process died within 1 second. exit.
y-ken commented 10 years ago

the error raised by Encryption not available on this event-machine on my environment which confirmed with strace command. According to this page, we have to install libssl-dev(debian) or openssl-devel(RHEL/CentOS) before install gem of eventmachine. ref. https://github.com/tweetstream/tweetstream/issues/133

So, would you please try like below?

$ sudo yum install openssl-devel
$ sudo /usr/lib64/fluent/ruby/bin/gem install eventmachine
inokappa commented 10 years ago

Hi, @y-ken san

Thank you very very much for giving me many useful advice!! I'll try later. Please just a moment ;-)

Thank you.

inokappa commented 10 years ago

Hi, @y-ken san

I tried version 0.2.4. That plugin is running normally !!

I tried following environment.

I tried following step.

  1. sudo apt-get install libssl-dev
  2. sudo /usr/lib64/fluent/ruby/bin/gem install eventmachine --no-ri --no-rdoc -V
  3. sudo /etc/init.d/td-agent configtest
  4. sudo /etc/init.d/td-agent restart
  5. confirm /var/log/td-agent/td-agent.log

Thank you so much for your good advice !!!

※日本語で失礼いたします

手元の環境で正常に動作することを確認致しました。 お忙しい中を色々とアドバイスを頂きまして本当に有難う御座いました。

y-ken commented 10 years ago

Great. I'm happy to hear working correctly. :+1:

勉強会等でお目にかかったときには是非ご挨拶させてください！

y-ken commented 10 years ago

README has updated to avoid confusing such as this problem. https://github.com/y-ken/fluent-plugin-twitter/commit/1e52723

inokappa commented 10 years ago

@y-ken san

Thank you for your reply!

13 日の Fluentd Casual Talk #3 に参加させて頂きます！ 是非、その際にご挨拶させて下さい！ 宜しくお願い致します！

y-ken commented 10 years ago

Sounds Great! :+1: I'm looking forward to seeing you.