No output from Twitter stream

[vshvedov]

I'm not sure is this a miss-configuration or something more, but I have to output from Twitter stream. Configs are:

• Ubuntu 14.04
• Ruby 2.1.2
• fluentd and fluent-plugin-twitter installed as gems

fluentd.config:

<source>
  type twitter
  consumer_key ...API key from app.twitter.com...
  consumer_secret ...API secret from app.twitter.com...
  oauth_token ...Access token from app.twitter.com...
  oauth_token_secret ... Access token secret from app.twitter.com...
  tag                 input.twitter
  timeline            sampling
</source>

<match input.twitter>
  type stdout
</match>

Then I start fluentd as fluentd -c fluentd.conf -o ./out.log -v

Here is the output:

</ROOT>
2014-06-09 05:33:04 -0400 [info]: fluent/engine.rb:101:block in configure: adding source type="twitter"
2014-06-09 05:33:05 -0400 [info]: fluent/engine.rb:117:block in configure: adding match pattern="input.twitter" type="stdout"
2014-06-09 05:33:05 -0400 [info]: plugin/in_twitter.rb:74:get_twitter_connection: twitter: starting Twitter Streaming API for sampling. tag:input.twitter keyword:ruby,rails,job
2014-06-09 05:33:53 -0400 [debug]: fluent/supervisor.rb:272:block in install_supervisor_signal_handlers: fluentd supervisor process get SIGINT
2014-06-09 05:33:53 -0400 [debug]: fluent/supervisor.rb:379:block in install_main_process_signal_handlers: fluentd main process get SIGINT
2014-06-09 05:33:53 -0400 [debug]: fluent/supervisor.rb:382:block in install_main_process_signal_handlers: getting start to shutdown main process
2014-06-09 05:33:53 -0400 [info]: fluent/engine.rb:230:run: shutting down fluentd
2014-06-09 05:33:53 -0400 [info]: fluent/supervisor.rb:236:supervise: process finished code=0
2014-06-09 05:35:47 -0400 [info]: fluent/supervisor.rb:219:supervise: starting fluentd-0.10.49
2014-06-09 05:35:47 -0400 [info]: fluent/supervisor.rb:321:read_config: reading config file path="fluentd.conf"
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-datacounter' version '0.4.3'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-elasticsearch' version '0.3.0'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-mongo' version '0.7.3'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-td' version '0.10.20'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-twitter' version '0.3.0'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluentd' version '0.10.49'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:91:configure: using configuration file: <ROOT>
  <source>
    type twitter
    consumer_key .......
    consumer_secret ...........
    oauth_token ..........
    oauth_token_secret ........
    tag input.twitter
    timeline sampling
    keyword ruby,rails,job
  </source>
  <match input.twitter>
    type stdout
  </match>
</ROOT>
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:101:block in configure: adding source type="twitter"
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:117:block in configure: adding match pattern="input.twitter" type="stdout"
2014-06-09 05:35:47 -0400 [info]: plugin/in_twitter.rb:74:get_twitter_connection: twitter: starting Twitter Streaming API for sampling. tag:input.twitter keyword:ruby,rails,job
2014-06-09 05:36:04 -0400 [debug]: fluent/supervisor.rb:272:block in install_supervisor_signal_handlers: fluentd supervisor process get SIGINT
2014-06-09 05:36:04 -0400 [debug]: fluent/supervisor.rb:379:block in install_main_process_signal_handlers: fluentd main process get SIGINT
2014-06-09 05:36:04 -0400 [debug]: fluent/supervisor.rb:382:block in install_main_process_signal_handlers: getting start to shutdown main process
........

I suppose here I have to see tweets stream, but nothing appears.

Something is wrong with output too.

Here is the config:

<source>
  type http
  port 8888
</source>

<match out.twitter>
  consumer_key ...
  consumer_secret ...
  oauth_token ...
  oauth_token_secret ...
  type twitter
</match>

Then I do:

curl http://localhost:8888/out.twitter -F 'json={"message":"hello from fluentd-twitter"}'

and output is:

2014-06-09 06:31:23 -0400 [warn]: fluent/engine.rb:156:rescue in emit_stream: emit transaction failed  error_class=NoMethodError error=#<NoMethodError: undefined method `update' for #<Twitter::Client:0x000000029a32b8>

Did I used Twitter's API/Access token and secrets correctly (Twitter's terminology differs, however I tried to swap API/ACCESS with no results - no errors, no output)?

Any ideas what is wrong?

[y-ken]

Hi, @vshvedov

Thank you for reporting issue. I have tested with latest version of ruby and gems (tweetstream and twitter). Then, I noticed there are problem on using current version of these gems.

So I will catch up the usage of gems and fixed to work correctly. If you could, would you please give me a pull-request for this issue?

[y-ken]

Hi, @vshvedov

I have released fluent-plugin-twitter v0.3.1 now. Would you please test it again which works correctly or not on your environment? https://rubygems.org/gems/fluent-plugin-twitter

---

If you have still problems, please tell me the output of gem list.

fluent-plugin-twitter ken$ ruby -v
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-darwin12.0]

fluent-plugin-twitter ken$ bundle exec gem list

*** LOCAL GEMS ***

addressable (2.3.6)
buftok (0.2.0)
bundler (1.6.2)
cookiejar (0.3.2)
cool.io (1.2.4)
daemons (1.1.9)
em-http-request (1.1.2)
em-socksify (0.3.0)
em-twitter (0.3.3)
equalizer (0.0.9)
eventmachine (1.0.3)
faraday (0.9.0)
fluent-plugin-twitter (0.3.1)
fluentd (0.10.51)
http (0.6.1)
http_parser.rb (0.6.0)
json (1.8.1)
memoizable (0.4.2)
msgpack (0.5.8)
multi_json (1.10.1)
multipart-post (2.0.0)
naught (1.0.0)
rake (10.3.2)
sigdump (0.2.2)
simple_oauth (0.2.0)
thread_safe (0.3.4)
tweetstream (2.6.1)
twitter (5.11.0)
yajl-ruby (1.2.1)

[y-ken]

Hi @vshvedov Closing this issue, ok? For these days, it does not cause a problem like issued.