URLFetch unreliability breaks hub content distribution

[GoogleCodeExporter]

It seems that the reference hub fails really often during distribution like 
this:

/work/push_events 200 5934ms 239cpu_ms 203api_cpu_ms 0kb AppEngine-Google; 
(+http://code.google.com/appengine)
10-02 02:28AM 58.751 /work/push_events 200 5934ms 239cpu_ms 203api_cpu_ms 0kb 
AppEngine-Google; (+http://code.google.com/appengine)

0.1.0.2 - - [02/Oct/2009:02:29:04 -0700] "POST /work/push_events HTTP/1.1" 200 
124 
"http://xxx.appspot.com/work/pull_feeds" "AppEngine-Google; 
(+http://code.google.com/appengine)" "xxx.appspot.com"
 I 10-02 02:28AM 59.344

1 more subscribers to contact for: topic = 
http://friendfeed.com/kugutsumen?format=atom, 
delivery_mode = normal
 D 10-02 02:28AM 59.345

Making call for RPC(urlfetch, Fetch, XXX, ..)

Waiting for RPC(urlfetch, Fetch, .., ..)
Could not deliver to target url XXX: Exception = None, status_code = 0
Could not deliver to target url XXX: Exception = ApplicationError(5,), 
status_code = unknown
Normal delivery done; 1 broken callbacks remain

** This is wrong, I checked the app engine logs on the subscriber and 204 
status code was sent 
correctly.

Sometimes it will complete after just one retry:
EventToDeliver complete: topic = http://friendfeed.com/kugutsumen?format=atom, 
delivery_mode = retry

but too often it will fail permanently after multiple retries. This is bad as 
we can't tell if a callback 
is working or not.

Another problem is on the subscriber side is that there is no way to detect 
this problem:

•  I 10-02 02:29AM 46.081 Attempting to create feed entry 
tag:friendfeed.com,2007:7b9e097b-e984-e903-bc90-73a7c853f75a (Artificial Heart 
Recipient 
Has No Pulse) for feed *7b0fa

•  I 10-02 02:29AM 46.227 Attempting to create feed entry 
tag:friendfeed.com,2007:e733e24e-
7441-75f6-3098-4a1fd8a5cb27 (s/any/in any/) for feed 
*7b0fa96012254a8638dd864661c8ff4

•  I 10-02 02:29AM 46.361 Attempting to create feed entry 
tag:friendfeed.com,2007:e1aa2325-
b407-9963-dae4-420f50723831 (Adium support for Twitter is pretty cool!) for 
feed *7b0

•  I 10-02 02:29AM 46.493 204 Finished processing 30 entries.

The subscriber returned a 204 status code... but the hub never saw it and it 
will keep retrying as 
if the delivery failed. There was no CPU or timeout warning on the subscriber 
side which also 
runs on app engine.

Thanks,
Anthony

Original issue reported on code.google.com by kugutsu...@gmail.com on 3 Oct 2009 at 5:03

[GoogleCodeExporter]

This is hitting the 5 second deadline (ApplicationError 5). I was going to 
raise the
default deadline up to 10 seconds, but subscribers should try to be faster than 
that
if possible, or else the Hub will retry content delivery.

Original comment by bslatkin on 19 Oct 2009 at 1:10

[GoogleCodeExporter]

Going to raise the deadline of the reference hub to 10 seconds.

Original comment by bslatkin on 22 Jan 2010 at 3:19

• Changed state: Accepted

[GoogleCodeExporter]

Fixed in r313. Will push to prod soon.

Original comment by bslatkin on 22 Jan 2010 at 3:51

• Changed state: Fixed