search

mccraigmccraig / twitter-streaming-client

a clojure client for the twitter streaming api, on top of twitter-api
26 stars 11 forks source link

retrieve-queues does not return results after app running for few days using start-twitter-stream #7

Open sandeepjagtap opened 9 years ago

sandeepjagtap commented 9 years ago

I see following logs after couple of days of application running twitter stream using start-twitter-stream.

15:17:38.712 [clojure-agent-send-off-pool-244] INFO twitter-streaming-client.impl - received chunk of length 2 15:17:38.712 [clojure-agent-send-off-pool-244] INFO twitter-streaming-client.impl - received 0 message lines after that, when app uses retrieve-queues, it returns nothing.

I see in logs [clojure-agent-send-off-pool-1] up to [clojure-agent-send-off-pool-255].

Any pointers that will help me debug this issue?

Ortuna commented 9 years ago

+1 same issue here, after a few days no errors, but no messages coming after.

mccraigmccraig commented 9 years ago

sorry : it's been an age since i've looked at this code

presumably you aren't getting anything from the record-exception-handler in the log ?

https://github.com/mccraigmccraig/twitter-streaming-client/blob/master/src/twitter_streaming_client/impl.clj#L228

sandeepjagtap commented 9 years ago

@mccraigmccraig thanks for reply. Yes, I do not see any messages from record-exception-handler in the logs. clojure-agent-send-off-pool-* keeps incrementing and after certain point no errors in log file, but messages do not come in.

MakotoTheKnight commented 9 years ago

To add a bit to this, I'm noticing that the server receives RST or FIN packets, and doesn't seem to restart of its own volition after receiving them. It's not that this would be an exceptional case; Twitter may occasionally drop the connections. Would seeing the packet stream of a few requests help to diagnose this issue? Further, does @sandeepjagtap experience the same thing? If you listen to the packets on your server - I used sudo tcpdump -K -n -i eth0 -s 1500 port not 22 and port not 53 -w - | tee /path/to/user/writable/log/packets.log | tcpdump -r - to write them to a file - do you see the symptoms exhibited by a RST/FIN packet?

Ortuna commented 9 years ago

After digging into http.async.client and com.ning/async-http-client. It could be:

If the streaming API disconnects via RST packets, http.async.client and in turn twitter-streaming-client would hang since it believes there is still an active connection.

astahlman commented 9 years ago

I'm seeing this issue, as well - no log messages, the stream just stops. I've reproduced the issue and managed to get a packet capture that confirms that the streaming API is closing the connection with a RST. Here's the relevant output from the packet capture:

14:38:50.548935 IP r-199-59-148-138.twttr.com.https > ${my-ec2-ip-address}.us-west-2.compute.internal.44828: Flags [F.], seq 28499974, ack 0, win 66, options [nop,nop,TS val 2633607053
ecr 441661854], length 0
14:38:50.589537 IP ${my-ec2-ip-address}.us-west-2.compute.internal.44828 > r-199-59-148-138.twttr.com.https: Flags [.], ack 28499975, win 7758, options [nop,nop,TS val 441662640 ecr 263
3607053], length 0
14:38:50.671852 IP ${my-ec2-ip-address}.us-west-2.compute.internal.44828 > r-199-59-148-138.twttr.com.https: Flags [P.], seq 0:37, ack 28499975, win 7758, options [nop,nop,TS val 441662
661 ecr 2633607053], length 37
14:38:50.711355 IP r-199-59-148-138.twttr.com.https > ${my-ec2-ip-address}.us-west-2.compute.internal.44828: Flags [R], seq 631764985, win 0, length 0

The stream stops right after the RST at 14:38:50.

The output of sudo netstat -t shows that the only active TCP connections are from my ec2 instance to New Relic and my ssh connection into ec2 - there is no connection to Twitter. This seems to indicate that Netty is handling the RST correctly and correctly dropping the connection.

Any suggestions as to how to proceed in light of this?