search

spyoungtech / grequests

Requests + Gevent = <3
https://pypi.python.org/pypi/grequests
BSD 2-Clause "Simplified" License
4.49k stars 331 forks source link

Calling map() with a bunch of requests (over 500) causes timeout error in individual requests, not sure how to catch these timeout exceptions #43

Closed codelucas closed 8 years ago

codelucas commented 10 years ago

When I attempt to call map with 500 or more requests, usually a few of my requests timeout (I set it to 7 seconds).

This is ok but the error messages display and i'm not sure how to catch these errors because all the work is done individually in the map() method from grequests.

Can I either catch these timeouts individually somehow, or somehow log all the failed requests and re-attempt them? What is usually the protocol when sending out a bunch of requests asynchronously with the requirement that they all return back.

netllama commented 10 years ago

I'm hitting a similar problem, although not quite on the same scale. Simply, I don't see how to catch the Timeout exception, and associate it with a specific URL.

codelucas commented 10 years ago

Hmm, since posting this issue I got around this problem by catching errors when a request fails and logging or printing out the failed url to try later.

Check out my modified version of grequests. I don't think it is worth sending a pull request for though.

codelucas commented 10 years ago

I was reading online, there is a good chance that the entire "Timeout" exception or whatever is a bit misleading. We are probably just getting rate limited by the target website!

DanMcInerney commented 10 years ago

A way to catch those exceptions wouldn't just be nice, it seems pretty necessary. That alone is likely to keep me from using this library since I'm scraping lots of sites that may or may not respond and I don't want the output spammed by exceptions.