search

mccraigmccraig / twitter-streaming-client

a clojure client for the twitter streaming api, on top of twitter-api
26 stars 11 forks source link

Make the agent fully reloadable #9

Open arichiardi opened 8 years ago

arichiardi commented 8 years ago

Hello @mccraigmccraig, I stumbled upon this very solid addition to twitter-api and started using it on a proof of concept.

I noticed that after I call the first time cancel-twitter-stream, I start receiving:

18:47:36.948 [clojure-agent-send-off-pool-5] WARN  twitter-streaming-client.impl - ignoring body from incomplete or failed request
18:47:36.949 [clojure-agent-send-off-pool-5] WARN  twitter-streaming-client.impl - protocol failure. failure-count 3. backing off 80000ms until 2016-06-29T01:48:56.948Z
18:47:36.951 [clojure-agent-send-off-pool-5] WARN  twitter-streaming-client.impl - http.async.client/Response{id: req-id__51333, status: #promise[{:status :ready, :val {:code 420, :msg "Enhance Your Calm", :protocol "HTTP/1.1", :major 1, :minor 1}} 0x7faf11cc], headers: #promise[{:status :ready, :val {:connection "close", :content-length "36", :content-type "text/html", :date "Wed, 29 Jun 2016 01:47:36 GMT", :server "tsa", :x-connection-hash "e62c00e283ba896df01f3349258783ff"}} 0x4d4de8ce], done: #promise[{:status :ready, :val true} 0x7863f850], error: #promise[{:status :pending, :val nil} 0x7f45a7e1]}

It looks like some state keeps the connection alive.

I am willing to investigate on this, the code is very readable (thanks!).

mccraigmccraig commented 8 years ago

investigation welcome - i'm not using twitter at the moment, and this project has received no love from me recently - i'm very happy for it to get some from others though :)

arichiardi commented 8 years ago

This is kind of weird, I define my state with mount's defstate ^{:on-reload :noop} and on clojure.tools.../refresh I don't see any tracing of the :start method. However, I still see:

12:24:19.951 [clojure-agent-send-off-pool-6] INFO  twitter-streaming-client.impl - starting request: twitter.api.streaming$statuses_filter@266f6b with params: (:oauth-creds #twitter.oauth.OauthCredentials{ ...yada yada...})

Which means that the request is sent somehow on the wire.

arichiardi commented 8 years ago

Ok the above problem might be in mount.

So the problem with reload-ability is actually in the fact that when I call cancel-twitter-stream and then create-twitter-stream too fast I receive a 420. Twitter sees this as a problem apparently:

Clients which break a connection and then reconnect frequently (to change query parameters, for example) run the risk of being rate limited.

So is my best bet to wait for 1 minute as suggested?

Back off exponentially for HTTP 420 errors. Start with a 1 minute wait and double each attempt. Note that every HTTP 420 received increases the time you must wait until rate limiting will no longer will be in effect for your account.