uwsgi behind ELB. OPTIONS response gets dropped, getting 502

[awesomescot-zz]

I'm running uwsgi in docker, behind an ELB. This is working for most requests, but when I make an OPTIONS cors request the ELB is dropping the response and returning a 502. For some reason everything works correctly when I run it with manage.py runserver, but not with uwsgi. I've used tshark to grab the responses and they look almost the same. Has anyone else run into this issue? Any help would be greatly appreciated. Thanks.

Response from uwsgi ( getting dropped by ELB ).

Hypertext Transfer Protocol
    HTTP/1.1 200 OK\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n]
            [HTTP/1.1 200 OK\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Version: HTTP/1.1
        Status Code: 200
        Response Phrase: OK
    Access-Control-Allow-Headers: accept, accept-encoding, authorization, content-type, dnt, origin, user-agent, x-csrftoken, x-requested-with\r\n
    Access-Control-Max-Age: 86400\r\n
    Vary: Origin\r\n
    Access-Control-Allow-Credentials: true\r\n
    Access-Control-Allow-Origin: https://staging.example.org\r\n
    Access-Control-Allow-Methods: DELETE, GET, OPTIONS, PATCH, POST, PUT\r\n
    Content-Type: text/html; charset=utf-8\r\n
    \r\n
    [HTTP response 1/1]
    [Time since request: 0.006864171 seconds]
    [Request in frame: 75]

Here is the response from runserver which successfully gets passed through the ELB.

Hypertext Transfer Protocol
    HTTP/1.0 200 OK\r\n
        [Expert Info (Chat/Sequence): HTTP/1.0 200 OK\r\n]
            [HTTP/1.0 200 OK\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Version: HTTP/1.0
        Status Code: 200
        Response Phrase: OK
    Date: Fri, 05 May 2017 15:06:29 GMT\r\n
    Server: WSGIServer/0.1 Python/2.7.13\r\n
    Access-Control-Allow-Headers: accept, accept-encoding, authorization, content-type, dnt, origin, user-agent, x-csrftoken, x-requested-with\r\n
    Access-Control-Max-Age: 86400\r\n
    Vary: Origin\r\n
    Access-Control-Allow-Credentials: true\r\n
    Access-Control-Allow-Origin: https://staging.example.org\r\n
    Access-Control-Allow-Methods: DELETE, GET, OPTIONS, PATCH, POST, PUT\r\n
    Content-Type: text/html; charset=utf-8\r\n
    \r\n
    [HTTP response 1/1]
    [Time since request: 0.001573862 seconds]
    [Request in frame: 100]

[funkybob]

I've run into various problems with ELB and uWSGI.

Unfortunately, the AWS team seems to think logging that there was a fault is enough, no need to log why they rejected the response!

What I found (and was added to the docs) is you must either have chunked encoding, or a Content-Length in the response.

See the comments here: http://uwsgi-docs.readthedocs.io/en/latest/HTTP.html#can-i-use-uwsgi-s-http-capabilities-in-production

[awesomescot-zz]

thanks for the response. You might be right. I thought I had added that, but looking at the response I don't see it. Still confused why runserver response is accepted, it doesn't have a content-length header either. Anyway, I had to get things working and ended up putting nginx in front anyway.

[funkybob]

It may also be a combination of this and keep-alive, which runserver does not support.

I believe it has something to do with ELB being certain where replies end, wanting "explicit over implicit", needing either an explicit content-length, or an explicit end of message state as given by chunked encoding.

[jchv]

I'm currently hitting this issue. I kind of figured it had to do with the abrupt end of the response since I noticed cURL 'assumes' close:

OK, so I understand this is mostly because ELB sucks. However, I'm kind of reliant on ELB. Is there anything I can do at the uWSGI level to prevent this? Perhaps some option that will insert a magic Content-Length: 0 on an empty response? I'll RTFM a couple more times...

For reference and for future Google searches: I hit this issue when using django-cors-headers with Django 1.11 under Amazon ELB with uWSGI's HTTP server.

edit: It appears you can successfully work around the issue by adding the http-keepalive and http-auto-chunked options. Probably a little overkill, but I'll take anything at this point. It seems autochunked does not take into effect unless you also specify http-keepalive, presumably because it doesn't do anything unless keepalive is enabled.

[funkybob]

Yeah, those and http-auto-gzip seem to all be tied together.

[jchv]

Just for reference, I actually continued having issues and ended up solving things in multiple ways.

First off, the uWSGI configuration that got me working was this:

[uwsgi]
http = :8080
http-keepalive = true
http-auto-chunked = true
add-header = Connection: Keep-Alive
module = [project].wsgi
static-map = /static=/tmp/static
enable-threads = true
workers = 4
die-on-term = true

route-run = chunked:
route-run = last:

I'm not sure whether it was add-header or the two route-runs that I needed, but after adding those two sections my problem is finally solved. I definitely needed it because Django's StreamingHttpResponse was not working, presumably because Django assumes it will be chunked by the WSGI server. I don't think this one is really Django's fault, since it seems from the end of a WSGI app, you can't really do chunked encoding.

However, I didn't figure this out immediately. Instead, I fixed the problem for only ordinary HttpResponses.

class EnsureContentLength:
    """
    Django middleware that ensures the Content-Length header is present on
    empty responses. ELB's Level 7 HTTP load balancer will drop some responses
    that have empty response bodies if you do not set the Content-Length
    header to zero.
    """

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)

        # Add content-length header for empty responses.
        if isinstance(response, HttpResponse):
            if not response.content:
                response['Content-Length'] = '0'

        return response

I'm still confused about a couple of things:

• Why didn't StreamingHttpResponse work to begin with? I don't have to set Content-Length for non-empty responses normally, so what made that different?
• What uWSGI settings are required for http-auto-chunked = true to work? It seems simply setting that alone doesn't work and I needed several other settings to actually get it to go.

But either way, I'm glad to have it working. I'm not sure if these limitations apply to the new "Application Load Balancers" because Kubernetes does not support them, but it's worth noting this only applies to the HTTP mode of ELB. This mode is desirable since it fills in the X-Forwarded-For and X-Forwarded-Proto, but it does break some things (like WebSockets) that ALB is supposed to work fine with.

I think maybe it'd be worth updating the documentation a bit to clarify a bit, at least w.r.t. how to enable http auto chunking.

[funkybob]

Yeah, I think I'll tackle some documentation updates on this when I get done documenting my last two features. :)

[medox]

thanks @jchv your comment was very helpful, we were facing the same issue using django-cors-headers with Django 1.10.4 under Amazon ELB.

[jchv]

Glad you figured out your problems. This is exactly why I go verbose on issue trackers :)

[kevin868]

I found http-keepalive = 1 is needed rather than true https://github.com/unbit/uwsgi/issues/2018

This fixed 502s for me, (and I do not have auto-chunked or route-run). Another key was using http mode, instead of http-socket which Uwsgi docs mention:

The http and http-socket options are entirely different beasts. The first one spawns an additional process forwarding requests to a series of workers (think about it as a form of shield, at the same level of apache or nginx), while the second one sets workers to natively speak the http protocol. TL/DR: if you plan to expose uWSGI directly to the public, use --http, if you want to proxy it behind a webserver speaking http with backends, use --http-socket.

https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html

[NickCellino]

@kevin868 I just wanted to say thank you for saving my sanity. This worked for me too!

I still don't fully understand the root of the problem unfortunately... but oh well!

[funkybob]

@kevin868 I just wanted to say thank you for saving my sanity. This worked for me too!

I still don't fully understand the root of the problem unfortunately... but oh well!

Oh, I think I ran into this some years ago ... it's because the --http-keepalive option specifies a timeout, not a boolean to enable the feature.