Support for streamed HTTP

[trott13]

While working with Server Sent Events using Tornado, I discovered, that (at least) the WSGIContainer in wsgi.py will set a Content-Length header field in any case.

However, this is not permitted, if a chunked or streamed HTTP result should be delivered. The Browser will interpret the set Content-Length as "this is no stream" and close the socket (which in turn kill the EventSource there and forces it to re-connect).

I changed the setting of Content-Length therefore into:

---

    # TRott, 12.7.2011: SSE and Webkit problem:
    # the response should only have a set Content-Length header field,
    # if it is not a HTTP-stream. In these cases HTTP is made of chunks
    # separated by double newlines (like our SSEvents...).
    # Therefore, if Content-Length is set, we are assumed
    # _not_ to stream...
    # We check the content-type for annoncing a stream here and switch of
    # the Content-Length header field, if it is:
    #
    isStream=False
    for (k,v) in headers:
        if k=="Content-Type" and v.endswith("stream"):
            isStream=True
            break

    if not isStream and "content-length" not in header_set:
        headers.append(("Content-Length", str(len(body))))

---

You probably should consider a more generalized solution to handle HTTP stream content in the future.

[bdarnell]

WSGIContainer is not suitable for streaming because it is single-threaded. Native tornado applications, on the other hand, can produce streaming output with RequestHandler.flush().

Also note that there is no special meaning for Content-Types that end with "stream", and responses that do not include Content-Length should generally use Transfer-Encoding: chunked.

[trott13]

Thanks, for the comment.

I use WSGIContainer, because I use Tornado driving a Django Web-Application. Can you suggest a better method for interfacing Tornado with Django?

I also added Transfer-Encoding: chunked to the header, resulting in Chrome trying to download something non existent. Conclusion: ServerSentEvents are not a "HTTP stream" in a regular fashion: both "Content-Length" and "Transfer-Encoding" are simply not working.

[bdarnell]

Why do you want to use both django and tornado? The usual reason to do this is to do any streaming/long-polling stuff as asynchronous tornado handlers; if you want to do streaming via django you'll have to use a multithreaded wsgi container instead.

It's not enough to just set the Transfer-Encoding: chunked header; you have to encode the data in chunks (see e.g. tornado.web.ChunkedTransferEncoding).

[peter-the-tea-drinker]

You can keep Tornado and Django at arms length. Set the Web Server to serve Django from some URLs, and Tornado from others. I'm no an expert, but I think it works like this: Nginx server (listening on port 80) forwards "example.com/app" to Apache on "127.0.0.1:8080", and forwards "example.com/async" to Tornado.