Twitter Logstash - undefined method `filter' for nil:NilClass

[Azlirn]

Hi all,

I am working on utilizing the ELK stack to do some live Twitter research and I am running into some issues when utilizing large tracks. I know by Twitter's standards, the track size has a limit of 400 terms with a byte size of each term between 1 and 60 bytes.

I have checked my "track" in the configuration file and I am using 220 unique terms.

I do notice that if I keep my "track" REALLY short (like less than ten terms) I don't seem to run into this issue.

So my issues are,

1. How do I fix the error?
2. This error doesn't seem to match the scenario, thoughts?
   • Version: 2.3.4
   • Operating System: OSX
   • Config File: http://hastebin.com/ovasusufos.coffee
   • Full Error: undefined method 'filter' for nil:NilClass {:exception=>#<NoMethodError: undefined method 'filter' for nil:NilClass>, :backtrace=>["/Users/.../logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.2/lib/logstash/inputs/twitter.rb:128:in 'run'", "/Users/.../logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:342:in ' 'inputworker'", "/Users/.../logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:336:in 'start_input'"], :options=>nil, :level=>:warn}

[andrewvc]

This is bizarre, somehow @stream_client is null here : https://github.com/logstash-plugins/logstash-input-twitter/blob/master/lib/logstash/inputs/twitter.rb

I'm not sure what's going on here, but this seems like a bug related to timing of some sort.

[andrewvc]

@Azlirn can you run this with debug logs and post those here? (be sure to scrub them of any sensitive data).

I spoke with @jordansissel and our money seems to be on something weird related to Logstash shutting down early.

[Azlirn]

@andrewvc

I will get this to you ASAP. Thanks for taking a look at this for me!

[ryumei]

I got similar error with logstash 5.0.0.

On start up

Sending Logstash logs to /opt/local/elastic/logstash-current/logs which is now configured via log4j2.properties.
[2017-03-08T00:29:00,376][INFO ][logstash.inputs.twitter  ] Starting twitter tracking {:track=>"engineer\\'s"}
[2017-03-08T00:29:00,376][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-03-08T00:29:00,403][INFO ][logstash.pipeline        ] Pipeline main started
[2017-03-08T00:29:00,594][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
[2017-03-08T00:29:01,739][WARN ][logstash.inputs.twitter  ] Twitter client error {:message=>"", :exception=>"Twitter::Error::Unauthorized", :backtrace=>["/opt/local/elastic/logstash-current/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:21:in `on_headers_complete'", "org/ruby_http_parser/RubyHttpParser.java:370:in `<<'", "/opt/local/elastic/logstash-current/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:16:in `<<'", "/opt/local/elastic/logstash-current/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-3.0.3/lib/logstash/inputs/twitter/patches.rb:31:in `stream'", "/opt/local/elastic/logstash-current/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-3.0.3/lib/logstash/inputs/twitter/patches.rb:58:in `request'", "/opt/local/elastic/logstash-current/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/client.rb:37:in `filter'", "/opt/local/elastic/logstash-current/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-3.0.3/lib/logstash/inputs/twitter.rb:138:in `run'", "/opt/local/elastic/logstash-current/logstash-core/lib/logstash/pipeline.rb:331:in `inputworker'", "/opt/local/elastic/logstash-current/logstash-core/lib/logstash/pipeline.rb:325:in `start_input'"], :options=>nil}

(same errors were continued)

then

[2017-03-08T00:30:35,996][WARN ][logstash.inputs.twitter  ] Twitter client error {:message=>"undefined method `filter' for nil:NilClass", :exception=>"NoMethodError", :backtrace=>["/opt/local/elastic/logstash-current/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-3.0.3/lib/logstash/inputs/twitter.rb:138:in `run'", "/opt/local/elastic/logstash-current/logstash-core/lib/logstash/pipeline.rb:331:in `inputworker'", "/opt/local/elastic/logstash-current/logstash-core/lib/logstash/pipeline.rb:325:in `start_input'"], :options=>nil}

(same errors were continued)

My config file is

input {
  twitter {
    consumer_key => "*********"
    consumer_secret => "*********"
    oauth_token => "*********"
    oauth_token_secret => "********"

    keywords => ["engineer\'s"]
  } 
}

The keywords (aka track) ["engineers"] work fine. However, ["engineer's"] and ["engineer\'s"] get error. on the other hand,

[gentunian]

In my case this is happening randomly when a new configuration is deployed via kibana Logstash -> Pipelines. Then, logstash can't start the pipeline looping in this error.

[dihiaselma]

I'm getting the same error. In my case, I've created a new twitter app (in may2022) with an elevated access => And I released that several changes are made to the creation of twitter app.

My pipeline is very simple: input { twitter { consumer_key => "40I2hjtut*****" consumer_secret => "y1AM***" oauth_token => "926076733***" oauth_token_secret => "CopTDMX0mbG****" keywords => ["engineer"] full_tweet => true type => "tweet" } } output { stdout{} }

I'm getting these errors.

Starting Logstash {"logstash.version"=>"7.11.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [mswin32-x86_64]"}
[2022-06-11T13:19:28,138][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2022-06-11T13:19:31,048][WARN ][logstash.inputs.twitter  ][main][a96d27becfc422b923a6e1a95c5c731bbd92ac2b0ea46872dbd8edc02622b758] Twitter client error {:message=>"", :exception=>Twitter::Error::Forbidden, :backtrace=>["C:/Users/Utilisateur/Desktop/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/twitter-6.2.0/lib/twitter/streaming/response.rb:24:in `on_headers_complete'", "org/ruby_http_parser/RubyHttpParser.java:370:in `<<'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/twitter-6.2.0/lib/twitter/streaming/response.rb:19:in `<<'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/twitter-6.2.0/lib/twitter/streaming/connection.rb:20:in `stream'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/twitter-6.2.0/lib/twitter/streaming/client.rb:119:in `request'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/twitter-6.2.0/lib/twitter/streaming/client.rb:38:in `filter'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-twitter-4.0.3/lib/logstash/inputs/twitter.rb:151:in `do_run'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-twitter-4.0.3/lib/logstash/inputs/twitter.rb:131:in `run'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/logstash-core/lib/logstash/java_pipeline.rb:405:in `inputworker'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'"], :options=>nil}

continue .....then this error

[2022-06-11T13:19:34,366][WARN ][logstash.inputs.twitter  ][main][a96d27becfc422b923a6e1a95c5c731bbd92ac2b0ea46872dbd8edc02622b758] Twitter client error {:message=>"undefined method `filter' for nil:NilClass", :exception=>NoMethodError, :backtrace=>["C:/Users/Utilisateur/Desktop/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-twitter-4.0.3/lib/logstash/inputs/twitter.rb:151:in `do_run'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-twitter-4.0.3/lib/logstash/inputs/twitter.rb:131:in `run'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/logstash-core/lib/logstash/java_pipeline.rb:405:in `inputworker'", "C:/Users/Utilisateur/Desktop/logstash-7.11.2/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'"], :options=>nil}

PS: when I use my old APP (that I created in 2018) => it works perfectly

SO what could be the issue ?

[andsel]

@dihiaselma seems like there is an incompatibility at API layer from your new app and the old one. The exception reported is Twitter::Error::Forbidden so I think it could be related to some of the authentication settings. If you are able to reproduce with a minimal test, please refer to the plugin's repository, this repository is intended for Logstash's core / general problem and not for specific plugins.

[andsel]

@dihiaselma https://github.com/logstash-plugins/logstash-input-twitter/issues/73 seems to be the same error

[andypiper]

In my case, I've created a new twitter app (in may2022) with an elevated access => And I released that several changes are made to the creation of twitter app.

Twitter apps created after April 29 2022 no longer have access to the v1.1 streaming API, even if you have Elevated access. Those endpoints are in the process of being retired.

[kares]

I do not think the original issue is relevant here - should be closed. there's a bug and the plugin repo: https://github.com/logstash-plugins/logstash-input-twitter/issues/74 I think given this is breaking for the Twitter input, it might get prioritized sooner ...

[viijaytdeshmukh]

Hi,I'm just trying the Twitter plugin from logstash documentation! Giving the same error as what you mentioned !

My questions are

What is nil in the twitter.rb file ? What are keywords ? How Twitter plugins actually work ?

Thanks!

[andsel]

Hi @viijaytdeshmukh please comment on the issue https://github.com/logstash-plugins/logstash-input-twitter/issues/74 if you have troubles related to the migration of the client library, so that we can track better. Thanks for your patience