search

y-ken / fluent-plugin-twitter

Fluentd Input/Output plugin to process tweets with Twitter Streaming API.
https://rubygems.org/gems/fluent-plugin-twitter
Other
53 stars 27 forks source link

undefined method `emit' for nil:NilClass #41

Closed y-ken closed 7 years ago

y-ken commented 7 years ago

Hi @okkez

I'm afraid do you know the solve this problem?

repeatable example

startup with following command. and exit wide ctrl-c.

$ bundle exec fluentd --conf fluentd_twitter.conf

error output

2017-04-03 11:47:57 +0900 [info]: #0 shutting down input plugin type=:twitter plugin_id="object:3fedc78fd44c"
2017-04-03 11:47:57 +0900 [info]: #0 shutting down input plugin type=:twitter plugin_id="object:3fedc78f50e4"
2017-04-03 11:47:57 +0900 [info]: #0 shutting down output plugin type=:twitter plugin_id="object:3fedc78e7d68"
2017-04-03 11:47:57 +0900 [info]: #0 shutting down output plugin type=:stdout plugin_id="object:3fedc6013abc"
2017-04-03 11:47:57 +0900 [info]: #0 shutting down output plugin type=:stdout plugin_id="object:3fedc6c97bbc"
2017-04-03 11:47:57 +0900 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::TwitterInput title=:in_twitter error_class=NoMethodError error="undefined method `emit' for nil:NilClass"
2017-04-03 11:47:57 +0900 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `emit' for nil:NilClass"
okkez commented 7 years ago

Oh! I missed this message. I'm checking this.

okkez commented 7 years ago

It seems that twitter gem's Twitter::Streaming::Client does not have API to close connection gracefully. So we cannot close connection to twitter.com while Fluentd shutdown sequence.

okkez commented 7 years ago

Maybe we need to implement Twitter::Streaming::Client#close and call it in Fluent::Plugin::TwitterInput#shutdown to stop firing new events from twitter stream:

def shutdown
  @client.close
  super
end
y-ken commented 7 years ago

Thank you for workaround solution! I'm surprised because this problem comes from inside ruby core.

okkez commented 7 years ago

:memo:

Using OpenSSL::SSL::SSLSocket#read_nonblock instead of #readpartial and implement Twitter::Client#close, Twitter::Streaming::Connection#close can resolve this problem.

See also following links:

y-ken commented 7 years ago

Thank you. I'll follow the updates.