buffer error (Zlib::BufError)

matthewmodestino commented 6 years ago

When using twurl to connect to the streaming twitter api to track a relatively rare keyword, I am receiving an buffer error and the command exits.

Expected behavior

I would expect the connection to remain open until something is received or until the command is interrupted purposefully.

Actual behavior

I run the twurl command and it runs for about 20 seconds, then quits with the following:

/usr/share/ruby/net/http/response.rb:357:in `finish': buffer error (Zlib::BufError)
    from /usr/share/ruby/net/http/response.rb:357:in `finish'
    from /usr/share/ruby/net/http/response.rb:262:in `ensure in inflater'
    from /usr/share/ruby/net/http/response.rb:262:in `inflater'
    from /usr/share/ruby/net/http/response.rb:274:in `read_body_0'
    from /usr/share/ruby/net/http/response.rb:201:in `read_body'
    from /home/splunker/.gem/ruby/gems/twurl-0.9.3/lib/twurl/request_controller.rb:14:in `block in perform_request'
    from /usr/share/ruby/net/http.rb:1419:in `block (2 levels) in transport_request'
    from /usr/share/ruby/net/http/response.rb:162:in `reading_body'
    from /usr/share/ruby/net/http.rb:1418:in `block in transport_request'
    from /usr/share/ruby/net/http.rb:1409:in `catch'
    from /usr/share/ruby/net/http.rb:1409:in `transport_request'
    from /usr/share/ruby/net/http.rb:1382:in `request'
    from /usr/share/ruby/net/http.rb:1375:in `block in request'
    from /usr/share/ruby/net/http.rb:852:in `start'
    from /usr/share/ruby/net/http.rb:1373:in `request'
    from /home/splunker/.gem/ruby/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:118:in `perform_request_from_options'
    from /home/splunker/.gem/ruby/gems/twurl-0.9.3/lib/twurl/request_controller.rb:13:in `perform_request'
    from /home/splunker/.gem/ruby/gems/twurl-0.9.3/lib/twurl/request_controller.rb:9:in `dispatch'
    from /home/splunker/.gem/ruby/gems/twurl-0.9.3/lib/twurl/abstract_command_controller.rb:7:in `dispatch'
    from /home/splunker/.gem/ruby/gems/twurl-0.9.3/lib/twurl/cli.rb:38:in `dispatch'
    from /home/splunker/.gem/ruby/gems/twurl-0.9.3/lib/twurl/cli.rb:21:in `run'
    from /home/splunker/.gem/ruby/gems/twurl-0.9.3/bin/twurl:4:in `<top (required)>'
    from /home/splunker/bin/twurl:23:in `load'
    from /home/splunker/bin/twurl:23:in `<main>'

Steps to reproduce the behavior

Run the following command, with any rare or low traffic term:

twurl -d track=rareterm -H stream.twitter.com /1.1/statuses/filter.json | jq .

One interesting thing I noted, if i search for a popular term...ie. "mlb", it streamed fine. Once I used ctrl+c to interrupt the command, i get the same zlib buffer error.

I tested on OSX 10.12.6 and on a centos 7 machine with the same result.

will test with curl as well, once I figure out how :)

taxen commented 5 years ago

+1 Even for not rare word, it happens sometimes. I run twurl every four hours to collect tweets having a word "bot." After an hour or two, sometimes (not always) it stops with the error.

pospinr1 commented 5 years ago

+1 Same here. CentOS 7, Piping into kafkacat ... Runs for an Hour or so ... twurl -v 0.9.3

Installed Packages Name : rubygems Arch : noarch Version : 2.0.14.1 Release : 34.el7_6

smaeda-ks commented 4 years ago

@matthewmodestino @taxen @pospinr1 I'm curious to know which Ruby version you all are (or were) using. This sounds like an issue from Ruby core/builtin Net::HTTP library... not twurl specific. I've been using twurl with at least Ruby 2.6+ and never seen this before. It may be worth testing it with a newer Ruby environment if you can.

smaeda-ks commented 4 years ago

No answer, closing this for the time being. Feel free to re-open this issue if you continue to discuss it. Thanks

nabeel-nazir-confiz commented 4 years ago

@smaeda-ks I am facing the same error with Ruby 2.0 I can not change the ruby version as my project has a lot of dependencies on it. Can you please share something why this happened and how can we resolve it? Thank you.

smaeda-ks commented 4 years ago

@nabeel-nazir-confiz I'm afraid that I can't tell you what's wrong there. The latest version of twurl only supports Ruby 2.4+ and we can't support you unless you're using one of those versions. Please consider updating to the latest version. If you can't update the global Ruby version, you still should be able to use a version manager such as rbenv.

twitter / twurl