Seeking input for refactoring exceptions

[jmathai]

See details for this issue on the wiki.

http://wiki.github.com/jmathai/twitter-async/seeking-input-for-exception-refactoring

[4s7r0]

I'm for it, although to be honest, I need those exceptions to trigger automatic throttling and other utilitarian tasks more than simple error handling.

[deplorableword]

Yep, this would be very handy

[jmathai]

@DieLaughing, Unfortunately, the API doesn't return limit specific error codes. You have to inspect the text that they return. A majority of the time I spend with the Twitter API is coming up with clever ways to deal with the throttling. In the version of EpiTwitter that I use for PubliciTweet we do in fact inspect the text to determine when we are being rate limited.

I could include that into the base EpiTwitter class which would check for some string or pattern and then throw an EpiTwitterOverLimitException or something. I don't like the idea of inspecting code to determine the actual cause of the error but it has worked so far and might be useful for devs.

In the end, dealing with being over limit has to be handled at the application layer. Will having an over limit specific exception help with that?

[4s7r0]

Yes. Error codes serve to distinguish the reason something isn't working, and I feel that an over limit response falls squarely in this category. My natural instinct is to define error handling by PHP class errors as well, but this is uniquely different. With minimal code you can open a robust, dynamic, and extensible way to automatically throttle requests and properly inform a site user of the situation. Developers desperately need integrated throttling with Twitter. I might be able to whip up an example that you could add to show you what I mean. I don't think it's worthy of a full code fork.

[jmathai]

Whip something up. If there's a way to do built in throttle management then I'm all for it. I'm going to add support to look at and inspect the headers Twitter sends back. That should help in handling different error cases more easily...and might help with what you're proposing as well.

[jmathai]

Unfortunately, authenticated POSTs to the api do not return a rate limiting status in the header. I've spoken with Doug from the api team in the past and this is by design.

While we can't include that it's fairly trivial to expose all of the header fields in the response and for unauthenticated GET requests I can use the rate limit information to more accurately throw an over limit exception.

To bad that's not possible for authenticated requests.

[jmathai]

jmathai@[~]: curl -u jmathai_test:jmathai_test -d "status=testing" -i http://twitter.com/statuses/update/jmathai_test.json
HTTP/1.1 200 OK
Date: Tue, 25 Aug 2009 00:15:40 GMT
Server: hi
Last-Modified: Tue, 25 Aug 2009 00:15:40 GMT
Status: 200 OK
ETag: "9295f22b446b3785a376cb9c57cfb715"
Pragma: no-cache
Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
Content-Type: application/json; charset=utf-8
Content-Length: 1053
Expires: Tue, 31 Mar 1981 05:00:00 GMT
X-Revision: d3260048b5ad7c18e1e74d5b16670905f02f4055
X-Transaction: 1251159340-52313-10991
Set-Cookie: lang=en; path=/
Set-Cookie: lang=en; path=/
Set-Cookie: _twitter_sess=BAh7CjoTcGFzc3dvcmRfdG9rZW4iLWY4MTJhYWQ3OWU1NzY1ZTlkNDY2OTE0%250ANGNiOGRhODY5NThmMmMwNWE6CXVzZXJpBMZdhAE6DGNzcmZfaWQiJTQ5NzMw%250AZTU3MGEwOGQyOWQ3MmJmYjdlOGRhOTAyMTdjOgdpZCIlMTgyYjE1ODU3OGZh%250AZTIxNTM2MTM5MWM0NjI2ODdiZGYiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xs%250AZXI6OkZsYXNoOjpGbGFzaEhhc2h7AAY6CkB1c2VkewA%253D--505f2f3d5fbcd4ded0dcdb41d99c80499d333cb2; domain=.twitter.com; path=/
Vary: Accept-Encoding
Connection: close

[4s7r0]

I haven't read your code well enough to know how to batch and cache multiple authenticated requests. Okay, I scanned it really quick and saw nothing. Right now I can't use it for anything more than walking a user through the initial OAuth dance. After that I have to write everything from scratch so I can use curl_multi_add_handle to keep a constant low number of requests pending and not overload their servers. Could you tell me if your code is capable of batching and caching individually authenticated requests? Because I have to be honest and say that the error handling is a very minor issue if I can't parallel requests at a controlled rate. Without batching and caching there's not much the error information is going to help you with anyway. "Twitter is down. Go away." would suffice for every failed request because no one would be using the site other than the developers.

[jmathai]

Yes, parallel calls are supported. http://wiki.github.com/jmathai/twitter-async#async

[smarcus]

After multiple authenticated requests to $twitterObj->get_usersShow (random, but in the range or 10-20+) I'm getting the Web Server exception:

Bad Request Your browser sent a request that this server could not understand. The number of requested header fields exceeds this server's limit.

I am not using epiTwitter asynchrously, I handle each request's returned values before issuing the next.

I presume this has something to do with front-end defense against attacks on the Twitter servers, and immediately die to avoid getting blacklisted. (At one point a bug did cause me to get half a dozen of these in a row, followed by a 503.)

Some low-level automatic throttling to remain very friendly with Twitter's current defenses would seem essential - it'd be very clumsy to implement at the application level, and I'd just be guessing, and handling it as an exception is like stopping when warning shots have already been fired into the air!

[jmathai]

If you're looping over code which "sometimes" throws exceptions then my hunch is that it's on Twitter's end. We do automatic throttling on our end for PubliciTweet because Twitter doesn't really provide rate limit information for authenticate calls (and don't plan to from my last email exchange with them). We see approx. 10% of calls returning non 200 responses. It's really high but I believe to use Twitter's APi you have to build your system to handle that. This includes handling exceptions by retrying certain calls and (in our case) adding our own throttling layer.

[smarcus]

It's not "sometimes", it's always, though the exact number of calls varies between 10 and 50. Sample code fragment:

foreach ($ids as $key => $id) { $userInfo = $twitterObj->get_usersShow (array('user_id' => $id)); try{ if ($userInfo->friends_count < 1000 && ...

Googling the "Bad Request" message it seems to come from Apache (though no documentation, just a list of bug reports), so this section of the Apache manual on DDOS attack protection tuning parameters is probably appropriate:

http://httpd.apache.org/docs/2.3/misc/security_tips.html

probably the LimitRequestFields Directive given that the error message is "The number of requested header fields exceeds this server's limit."

[jmathai]

Added a new API for the library. Adding versioning support. Includes oauth debugging parameter. closed by c653259a0c8448d92ba27d4d431b53ccf3dcf0a8 closed by c653259a0c8448d92ba27d4d431b53ccf3dcf0a8 closed by c653259a0c8448d92ba27d4d431b53ccf3dcf0a8 closed by c653259a0c8448d92ba27d4d431b53ccf3dcf0a8