tornadoweb / tornado

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.
http://www.tornadoweb.org/
Apache License 2.0
21.69k stars 5.5k forks source link

exception in httpclient on certain valid, HTTP-200 server responses for unknown reason #306

Closed mlogan closed 13 years ago

mlogan commented 13 years ago

If you run the following code:

#!/usr/bin/python2.6
from tornado import httpclient, ioloop
url = 'http://api.imgur.com/2/image/WMn80.json'

def callback(response):
    print response.body

http = httpclient.AsyncHTTPClient()
http.fetch(url, callback=callback, validate_cert=False, request_timeout=10.0)
ioloop.IOLoop.instance().start()

You get the following exception:

None
WARNING:root:uncaught exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/tornado/simple_httpclient.py", line 259, in cleanup
    yield
  File "/usr/lib/python2.6/site-packages/tornado/stack_context.py", line 183, in wrapped
    callback(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/tornado/simple_httpclient.py", line 341, in _on_chunk_length
    self._on_body(b('').join(self.chunks))
  File "/usr/lib/python2.6/site-packages/tornado/simple_httpclient.py", line 331, in _on_body
    self.callback(response)
TypeError: 'NoneType' object is not callable

However, there doesn't appear to be anything wrong with the imgur webserver. The request I paste into nc is the exact request that the tornado httpclient generates.

; nc api.imgur.com 80
GET /2/image/WMn80.json HTTP/1.1
Host: api.imgur.com
Accept-Encoding: gzip

HTTP/1.1 200 Ok
Server: nginx
Date: Mon, 18 Jul 2011 17:38:47 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
Set-Cookie: IMGURSESSION=1suc7a16qoc7k061nai8echn50; path=/; domain=.imgur.com
Access-Control-Allow-Origin: *
Pragma: public
Cache-control: private
Expires: -1
X-RateLimit-Limit: 500
X-RateLimit-Remaining: 488
X-RateLimit-Reset: 1311013013
Set-Cookie: SERVERID=www1; path=/

1a0
{"image":{"image":{"title":null,"caption":null,"hash":"WMn80","datetime":"2011-07-17 05:09:49","type":"image\/jpeg","animated":"false","width":1598,"height":1201,"size":119551,"views":372682,"bandwidth":44554505782},"links":{"original":"http:\/\/i.imgur.com\/WMn80.jpg","imgur_page":"http:\/\/imgur.com\/WMn80","small_square":"http:\/\/i.imgur.com\/WMn80s.jpg","large_thumbnail":"http:\/\/i.imgur.com\/WMn80l.jpg"}}}
0

I have verified that the fetch code I provided above works with URLs on other servers - it even works with other imgur URLs, such as http://imgur.com/WMn80. imgur appears to return a very similar response to that URL - same Transfer-Encoding, same Server, same Connection headers, so I have no clue what could be going wrong in the former case.

Thanks, Mark

bdarnell commented 13 years ago

I think this is basically a timing issue related to receiving multiple chunks in the final packet, in which case the connection-closed callback is getting run before all of the buffered data has been processed. The close callback needs to wait until the buffers are empty (and/or we need a way to run read callbacks against all the buffered data in a single ioloop iteration).