search

fennb / phirehose

PHP interface to Twitter Streaming API
709 stars 189 forks source link

Phirehose Suddenly Stopped Working #105

Closed DedicatedManagers closed 8 years ago

DedicatedManagers commented 8 years ago

I've been running Phirehose for a long time now with very few issues. I don't think I've touched my script for over 6 months. Just a few days ago it quit working. I tried restarting and this is the error message I get. I downloaded the latest revision and its still not working. Anyone else having issues, or can you help me diagnose the error below? Obviously I'm being blocked as unauthorized, but I logged into apps.twitter.com and verified my consumer key & consumer secret haven't changed.

ERROR: HTTP failure 1 of 20 connecting to stream: HTTP ERROR 401: Authorization Required (<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>\n<title>Error 401 Unauthorized</title></head><body><h2>HTTP ERROR: 401</h2><p>Problem accessing '/1.1/statuses/filter.json'. Reason:<pre> Unauthorized</pre></body></html>). Sleeping for 10 seconds.

tamirvs commented 8 years ago

Had the same issue just now. I've regenerated the access token and restarted my VM (VirtualBox) and now it's working again. Not sure what it was..

DedicatedManagers commented 8 years ago

tamirvs - thanks for the reply... I regenerated my OAUTH_TOKEN and OAUTH_SECRET and I'm still not having any success.

fennb commented 8 years ago

This can also happen if you try to log in with more than 1 account at once (eg: Development environment & production) or if you try to reconnect too fast (eg: bug in script that causes it to auto-reconnect).

Additionally, there's a small chance your account may be banned for some reason. If you can create a new account with new credentials and that works, then the banning would be your problem.

DedicatedManagers commented 8 years ago

fennb - Thanks very much for the reply. I do have another script that uses the same app account (ie. same OAUTH_TOKEN & OAUTH_SECRET) but it doesn't use phirehose and is only used when I manually take an action, so the other script is not connecting when I try to re-start my phirehose script which is continually unable to connect at this point.

The other script that uses the same account is working fine, so I don't think my account is banned, unless the account can get banned from using the services that phirehose connects to (ie streaming).

Just to be sure... I'll try making another app account and see if it solves the issue.

DedicatedManagers commented 8 years ago

I created a new developer app on apps.twitter.com.

From the "Keys and Access Tokens" tab I retrieved my "Consumer Key (API Key)" which I put in phirehose constant TWITTER_CONSUMER_KEY and my "Consumer Secret (API Secret)" which I put in the phirehose constant TWITTER_CONSUMER_SECRET.

Then I ran my login script to give the new app I created access to the twitter account I used to create the new app. I grabbed the oauth_token & oauth_token_secret stored from the login process and put them in the phirehose constants OAUTH_TOKEN & OAUTH_SECRET.

I'm still getting the same message as above. I even tried doing the login process with another separate twitter account and using the oauth_token & oauth_token_secret for that connection and still got the same error message above.

Any further thoughts?

fennb commented 8 years ago

Hmmm, very strange indeed. There's a small chance your IP/whole account has been banned from the streaming API.

You should try connecting from a different IP address, or create an entirely new twitter account and connecting.

One of those two things theoretically should work, or the world has gone crazy :)

oneafrikan commented 8 years ago

here in the UK, the world has gone crazy :-/

PS. Using Phirehose to consume #referendum tweets, sitting at abt 70m datapoints atm, and still going strong. All being stored on a little Dell T110 running Turnkey Linux VM. Next up - NLP ;-)

Too brief? Here's why! http://emailcharter.org

On 3 July 2016 at 08:06, Fenn Bailey notifications@github.com wrote:

Hmmm, very strange indeed. There's a small chance your IP/whole account has been banned from the streaming API.

You should try connecting from a different IP address, or create an entirely new twitter account and connecting.

One of those two things theoretically should work, or the world has gone crazy :)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fennb/phirehose/issues/105#issuecomment-230139187, or mute the thread https://github.com/notifications/unsubscribe/AG0CMHoDOWJ9tPorlRhxawpm_tjAm_Ooks5qR19ZgaJpZM4JAYgO .

DedicatedManagers commented 8 years ago

Well I tried another server (using the same api account) and its now working. So I guess they are blocking the IP of the server that can't connect for some reason.

Its strange though.. I'm getting responses (ie the "Unauthorized" mentioned above and shown again below) and the documentation (see far below) says that if I'm blocked I won't get any response at all. Does Phirehose output "Unauthorized" if it can't even connect to the server? It looks like the error below is the header of the response from the twitter api:

Here is the error again:

ERROR: HTTP failure 1 of 20 connecting to stream: HTTP ERROR 401: Authorization Required (<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>\n<title>Error 401 Unauthorized</title></head><body><h2>HTTP ERROR: 401</h2><p>Problem accessing '/1.1/statuses/filter.json'. Reason:<pre> Unauthorized</pre></body></html>). Sleeping for 10 seconds.

I don't know why I'd get banned. I just have the one phirehose script running and its a very simple filter for a couple keywords.

Really, I'm just banned from the streaming api... I tested and I can make calls to the REST api to do functions like retweet, favorite, tweet, etc.

Very Strange. tc

From the https://dev.twitter.com/faq:

Being banned or blacklisted means the Twitter APIs will not respond to requests you make to them. You know if you have been blacklisted because the APIs will not respond to you at all. If this happens to you the first thing to do is stop any requests your application is making. Then see if you can reach other URLs using a command line tool like curl. If you can access other URLs but not Twitter, you should login to Twitter.com and then file a ticket with our support team. The support team will then be in contact about next steps. Twitter provides APIs for free and with no guarantees of service availability. This means we may take steps to ban or block any account, IP, or range of IPs that might be harming our ability to provide Twitter in a timely and reliable way. The best way to avoid being blacklisted is to pay attention to the remaining API requests you are allowed to make, and to handle errors appropriately. Handling errors appropriately means reducing your request frequency (throttling) or stopping requests until you can identify why the request failed. If you have sufficiently diagnosed your network connectivity and believe your IP may have been blacklisted, please post to the discussion forums.. Provide as much detail as possible about which steps you’ve taken to diagnose.

fennb commented 8 years ago

Might be worth asking what's happening on the Twitter developers list.

Well, glad you worked it out!

DedicatedManagers commented 8 years ago

Thanks for your suggestions, which ended up helping. Here is my post over at twitter developer's forum: https://twittercommunity.com/t/being-blocked-with-unauthorized-but-only-from-one-ip/70072

Haven't received a response. Let me know if you know of a better place to post. tc

emahieu commented 6 years ago

For anybody encountering the same issue: check your server time, if it's off by a couple of minutes the "unauthorized" message will appear. That fixed it for me.

mandys commented 6 years ago

The solution mentioned by @emahieu worked for me. On centos for some reason ( yet to figure out ) clock was 5 minutes behind ( after never having that issue in years )

Running hwclock --debug ( hardware clock ) fixed the time mismatch and then twitter stream started working again.