Zlib._handle.onerror unexpected end of file

[Brideau]

I am receiving the same error as mentioned in issue https://github.com/ttezel/twit/issues/269, but the solution mentioned there does not fix the problem for me.

I'm using twit version 2.2.4, with the following configuration:

const keys = {
    consumer_key : process.env.TWITTER_CONSUMER_KEY,
    consumer_secret : process.env.TWITTER_CONSUMER_SECRET,
    access_token : process.env.TWITTER_ACCESS_TOKEN,
    access_token_secret : process.env.TWITTER_ACCESS_TOKEN_SECRET,
    timeout_ms: 60*1000,
  };

I've tried it both with and without the trailing comma at the end. It works for an extended period of time (usually a day or two), but then fails abruptly with this error:

events.js:160
    throw er; // Unhandled 'error' event
    ^

Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)

The tweet emitter is started and consumed using the following functions (I was testing different geographic areas, hence the geo filter that covers the whole planet):

const client = new Twitter(keys);
const twitEmitter = client.stream('statuses/filter',
  {locations: '-180,-89,180,89'});

twitEmitter.on('tweet', (tweet) => {
  filterForwardLog(tweet);
});

twitEmitter.on('disconnect', (disconnectMessage) => {
  winston.log('warning', 'Connection terminated: ');
  reject(disconnectMessage);
});

twitEmitter.on('connect', () => {
  winston.log('info', 'Connection attempted.');
});

twitEmitter.on('connected', () => {
  winston.log('info', 'Connection successful.');
});

twitEmitter.on('reconnect', (req, res, interval) => {
  winston.log('notice', `Reconnecting in ${interval / 1000.0} seconds`);
});

twitEmitter.on('warning', (warning) => {
  winston.log('warn', warning);
});

[olalonde]

Same thing here. This bug alone accounts for like 90%+ of my error logs, pretty annoying.

{ Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17) errno: -5, code: 'Z_BUF_ERROR' }
error:  Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)

[olalonde]

Created an issue on node (https://github.com/nodejs/node/issues/8701) just in case. I reviewed the twit code and it doesn't seem to do anything out of ordinary with the zlib streams. And if it was just an error that was thrown when a connection closes early or timeout, I suppose googling for that error would bring a lot more results (but it doesn't).

[chasset]

Hello, I have the same Z_BUF_ERROR. I would like to share a clue of the reason and the solution:

Twitter sends tweets in chunks. Sometimes, some chunks get lost. This throws an error when zlib tries to look for an end of the tweet. A correct behaviour for Twit would be to stay connected and to wait for the next good tweet.

I discover this developing in Perl my own library to do what Twit does.

What do you think about it?

[hoodsy]

Anyone have a temporary workaround or fix? This is basically shutting down my application.

I haven't recently changed versions of node or twit, but this error has randomly come around...

[ioRekz]

Yes same here, started since few days

[hoodsy]

@ioRekz do you remember changing anything around the time this happened? Could this be a change on Twitter's end with the streaming API?

[CTassisF]

This error always existed for me and it was quite intermittent, but today it got worse. Twitter Streaming API is closing too much connections today (maybe due to DNS problems they had with Dyn), you can see that yourself using cURL to stream (the TCP session should disconnect in few seconds). The problem with ttezel/twit is that it can't handle connections being abruptly closed by Twitter streaming endpoint. Instead of crashing, ttezel/twit needs to be able to handle these events and try to reconnect streams incrementally, but I do not know how to do that :/

[hoodsy]

@CTassisF my connection is polling, so it will reconnect after a short period of time. Must be related to the DNS attack on Dyn, as you say.

[CTassisF]

This issue might be related to NodeJS. I was using NodeJS 6.9.1 from nodesource/distributions but I decided to abandon it and try to use the NodeJS 4.6.0 from Debian Stretch official mirror. With NodeJS 4.6.0 the issue is gone.

[austinhuang0131]

Same here, started today.

[olalonde]

@CTassisF I tried opening an issue https://github.com/nodejs/node/issues/8701 but didn't have time to write some code to reproduce. I'm using Node v6. Maybe https://github.com/ttezel/twit/pull/304 will magically fix the bug? 😜

[ioRekz]

Who is using heroku ? I don't seem to have any error when starting the server locally.

[ioRekz]

@hoodsy Ho thought I answered but no, I'm sure I didn't change anything

[ioRekz]

@CTassisF I don't have the same behavior. It's almost every favorite and track events in my case that raise this error

[olalonde]

Not Heroku but Docker

[KManu]

Started using twit today, getting the same issue here.

[vanita5]

I get the same error since today.

Stream error: undefined
{ Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17) errno: -5, code: 'Z_BUF_ERROR' }

[calbertts]

Using NodeJS 4.6.0 the issue is gone, not sure if it has something to do with it.

[eduardbosch]

Like @calbertts & @CTassisF I confirm that issue is gone in NodeJS 4.6.0

[SeanEire]

Node 4.6.1 resolves this for me also.

[ioRekz]

Same, does anyone has an idea of what's different on 4.6.0 / 4.6.1 in regard to this issue ?

[CTassisF]

I commented on nodejs/node#8701 saying that there is no issue on 4.6.0/4.6.1, but I don't know how to troubleshoot this any further.

[CTassisF]

Even running Node 4.6.1 I had some problems today. Again. Did you see the same?

I found out that Twitter has a community site with real Twitter staff there. They are aware of issues with streaming endpoints. You can see it here: https://twittercommunity.com/t/issues-reported-with-streams-since-10-21/76429

I also noticed that the IPs for all streaming endpoints have changed and after that the issue is gone again. Did not try with Node 6.9.1 yet, but I think they might have fixed it for good. I posted about this in the community: https://twittercommunity.com/t/issues-reported-with-streams-since-10-21/76429/5?u=ctf

[super-ienien]

Hi guys. I've got this issue since several months. It became even more frequent since the DDOS attack. What I did to solve the problem was to fork twit and disable gzip for the stream API. Now it works perfectly

[conradoplg]

This seems related to the SO question which mentions this change introduced in node.js 5, though I haven't examined the details

[chasset]

@super-ienien How have you done that?

[super-ienien]

I did that like this : https://github.com/super-ienien/twit/commit/982d227347256a958f09f6c1062437a5cdb91081

just look at the modification inside : streaming-api-connection.js and twitter.js the modifications in others files are not related to this issue

But do not use my fork because it's not up to date with twit last version

[mhoanghold]

@whitesheep I used your fork and separately, just your changes, but I am still getting Z_BUF_ERROR and Z_DATA_ERROR

[ioRekz]

I didnt change anything for sure. It's running on heroku and I didnt updated the app for a while

On Oct 22, 2016 7:14 AM, "César de Tassis Filho" notifications@github.com wrote:

This issue might be related to NodeJS. I was using NodeJS 6.9.1 from nodesource/distributions https://github.com/nodesource/distributions but I decided to abandon it and try to use the NodeJS 4.6.0 from Debian Stretch official mirror. With NodeJS 4.6.0 the issue is gone.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ttezel/twit/issues/300#issuecomment-255493368, or mute the thread https://github.com/notifications/unsubscribe-auth/AAqChrG9u895k6h7wJtShChGu3KkEOcbks5q2VVagaJpZM4J97MO .

[SeanEire]

Has anyone tried using Node v7 to see if this makes any difference? I haven't had a chance to try it out yet.

[izumskee]

@SeanEire for me this bug is gone with Node v7

[CTassisF]

According to Twitter staff (and other reports) the issue with Twitter Streaming API that was triggering this zlib.js crash has been fixed. There is a post on Twitter Developer Forum regarding this issue. I am back running Node.js v7.0.0 from NodeSource and everything is looking good so far.

[olalonde]

Yep, haven't seen this issue in my logs for the past >24h. (node v6)

[olalonde]

Had this happen again today, but it's definitely not often as before.

[chasset]

The error from last month is back.

I launch a statuses/sample stream and I get tweets during one minute. Then this error is emitted.

Error: {
  "errno": -5,
  "code": "Z_BUF_ERROR"
}

This stops the reception of tweets. After a minute, twit try to reconnect normally. This cycle repeats itself forever: one minute of tweets and one minute of silence.

• Version: node v7.2.0 – Twit 2.2.5
• Platform: Linux 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[Brideau]

@chasset Are you still getting this error running 4.6.x? I ended up switching libraries, but the other library has the same problem (only it fails silently...which is worse). If running it on an older version of Node works, that's good enough for me for now.

[conradoplg]

I've been running for weeks with my PR #309, no problems so far

[olalonde]

@ttezel can we merge this plz? :)

[chasset]

@Brideau Yes, with node 4.6.2, I still have this problem

[chasset]

@Brideau I try on another machine (macOS Sierra) with node 6.9 but I get the same error periodically (1-3 minutes).

But, it happens only on heavy streams like:

• statuses/filter with a bounding box of [-180,-90,180,90] or
• statuses/sample.

[pantchox]

Hi, Consider this code snippet

var stream = T.stream('statuses/sample')
stream.on('tweet', function (tweet) {
    console.log(tweet);
});

Another example to receive this error if i simply unplug my wifi card.

is there something like a try catch that can be implemented to try to bypass this issue? Thanks.

[pantchox]

Sorry on my previous post, if you do

stream.on('error', (err) => {
  console.log('error!', err);
});

the program is not exiting which is perfect.

[cstephen]

What @pantchox says works for me, too. I was also able to replicate the error by cutting off internet access (disabling my virtual machine's network adapter) while my script was running. Here's my understanding of what's going on:

Abruptly cutting off the Twitter stream emits a gunzip "error" event that's handled by the following listener in streaming-api-connection.js (at line 138 currently):

gunzip.on('error', function (err) {
  self.emit('error', err);
})

The code above re-emits the "error" event. You need to add a listener to your script to handle it, like @pantchox said:

stream.on('error', (err) => {
  console.log('error!', err);
});

If you don't handle the "error" event in your script, Node.js will handle it for you by exiting. See https://nodejs.org/api/events.html#events_error_events

If an EventEmitter does not have at least one listener registered for the 'error' event, and an 'error' event is emitted, the error is thrown, a stack trace is printed, and the Node.js process exits.

[kud]

The workaround works here for me too; but is it normal? Is it happening because of network troubles?

[devinaction]

Restarting the app or checking for a previously running instance of the app helps.