Error during the websocket handshake: 404

[MoncefB]

Hello,

First of all, kudos for everything <3

Please, I have basically exhausted all the already existing topics (previously opened issues on Github, questions on StackOverflow, etc.) about the error message "WebSocket connection to 'wss://www.mbiaz.org/ws/v1.0/office/' failed: Error during WebSocket handshake: Unexpected response code: 404"

The weird thing is that I moved all the traffic to Daphne (I used to rely on Gunicorn but not anymore) for both HTTP requests and sockets. My Django app is definitely running as ASGI. These are my dependencies:

aioredis==1.2.0 amqp==1.4.9 analytics-python==1.2.5 anyjson==0.3.3 apiai==1.2.3 asgiref==2.3.2 async-timeout==2.0.1 atomicwrites==1.2.1 attrs==18.2.0 autobahn==19.1.1 Automat==0.7.0 Babel==2.6.0 billiard==3.3.0.23 bleach==3.1.0 boto==2.48.0 boto3==1.3.1 botocore==1.4.93 bz2file==0.98 celery==3.1.25 certifi==2018.11.29 channels==2.1.6 channels-redis==2.3.2 chardet==3.0.4 Click==7.0 constantly==15.1.0 contractions==0.0.17 cycler==0.10.0 daphne==2.2.4 decorator==4.3.2 defusedxml==0.5.0 Django==2.0.1 django-axes==2.1.0 django-celery==3.2.1 django-constance==2.0.0 django-debug-toolbar==1.8 django-extensions==1.9.1 django-localflavor==1.3 django-nose==1.4.4 django-picklefield==2.0 django-webpack-loader==0.5.0 dnspython==1.16.0 docutils==0.14 emoji==0.3.9 entrypoints==0.3 enum34==1.1.6 Flask==1.0.2 Flask-Caching==1.4.0 flower==0.9.1 freezegun==0.3.8 fuzzywuzzy==0.16.0 gensim==3.6.0 graphviz==0.8 gunicorn==19.6.0 hashids==1.1.0 hiredis==1.0.0 humanize==0.5.1 hyperlink==18.0.0 idna==2.6 incremental==17.5.0 ipwhois==1.0.0 ipykernel==5.1.0 ipython==5.2.0 ipython-genutils==0.2.0 ipywidgets==7.4.2 itsdangerous==1.1.0 Jinja2==2.10 jmespath==0.9.3 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.2.4 jupyter-console==6.0.0 jupyter-core==4.4.0 Keras==2.0.6 kombu==3.0.37 MarkupSafe==1.1.0 matplotlib==2.0.2 mistune==0.8.4 more-itertools==5.0.0 msgpack==0.5.6 nbconvert==5.4.0 nbformat==4.4.0 nltk==3.4 nose==1.3.7 notebook==5.7.4 numpy==1.16.0 olefile==0.46 pandas==0.20.1 pandocfilters==1.4.2 pathlib2==2.3.3 pexpect==4.6.0 pickleshare==0.7.5 Pillow==4.2.1 pkg-resources==0.0.0 plaid-python==2.3.3 pluggy==0.8.1 prometheus-client==0.5.0 prompt-toolkit==1.0.15 psycopg2==2.7.3.2 ptyprocess==0.6.0 py==1.7.0 pycorenlp==0.3.0 pydot==1.2.3 Pygments==2.3.1 PyHamcrest==1.9.0 PyJWT==1.7.1 pyparsing==2.3.1 PySocks==1.6.8 pytest==4.0.1 python-dateutil==2.5.3 python-Levenshtein==0.12.0 pytz==2018.9 PyYAML==3.13 pyzmq==17.1.2 qtconsole==4.4.3 raven==6.6.0 redis==2.10.5 requests==2.18.4 scaleapi==0.1.8 scikit-learn==0.18.2 scipy==1.2.0 Send2Trash==1.5.0 simplegeneric==0.8.1 singledispatch==3.4.0.3 six==1.12.0 sklearn-pandas==1.5.0 smart-open==1.8.0 sqlparse==0.2.4 terminado==0.8.1 testpath==0.4.2 Theano==1.0.4 tornado==4.2 traitlets==4.3.2 twilio==6.10.0 Twisted==18.9.0 txaio==18.8.1 ua-parser==0.7.3 urllib3==1.22 uszipcode==0.1.3 wcwidth==0.1.7 webencodings==0.5.1 Werkzeug==0.14.1 widgetsnbextension==3.4.2 zope.interface==4.6.0

I am confident about my routing: `from channels.auth import AuthMiddlewareStack from channels.routing import ProtocolTypeRouter, URLRouter from django.conf.urls import url

from account.consumers import PlaidLinkConsumer

application = ProtocolTypeRouter({

(http->django views is added by default)

'websocket': AuthMiddlewareStack(
    URLRouter([
        url(r'^ws/v1.0/office/$', PlaidLinkConsumer),
    ])
),

}) `

I'm using Daphne through Supervisor: `[fcgi-program:daphne] socket=tcp://localhost:8000

directory=/home/ubuntu/src/charlie

command=/home/ubuntu/Envs/charlie_env/bin/daphne -u /tmp/daphne/daphne%(process_num)d.sock --fd 0 --access-log - --proxy-headers charlie_web.charlie_web_project.asgi:application

numprocs=2

process_name=daphne%(process_num)d

autostart=true autorestart=true

stdout_logfile=/var/log/daphne.log redirect_stderr=true

environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8,PYTHONPATH="/home/ubuntu/src/charlie/charlie_web/" `

My OS is Ubuntu 16, and I'm using Nginx. I followed the Channels docs about Deployment, and I did add the lines about HTTP 1.1 and the connection upgrade:

` upstream channels-backend { server localhost:8000; }

server { listen 80; server_name mbiaz.org;

access_log /var/log/nginx/access.log custom_log;

location /static/ {
    alias /home/ubuntu/www/static/;
}

location / {
    try_files $uri @proxy_to_app;
}

location @proxy_to_app {
    proxy_pass http://channels-backend;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $server_name;
}

} `

The HTTP part works well, when I visit the page supposed to use websockets. The page is rendered properly, but the socket part fails with the 404... When I look at the Daphne log, I get: 2019-01-25 08:35:07,904 WARNING Not Found: /ws/v1.0/office/

What is going wrong? :(

[carltongibson]

Hi @MoncefB.

There's essentially no way of telling what your problem is from the information you've provided. It's the entire stack (!). You'll need to break it down.

Presumably it works locally using runserver? So can you run Daphne locally and test that? Then can you test directly requesting against Daphne running under supervisor (rather than via the proxy)? Then via the proxy? At what layer does it stop working? Can you add logging and check exactly what's being passed/received? You say you're confident of the routing, well, can you put a breakpoint in URLRouter and check it's being hit? And so on...

Sorry I can't say more, but there's too much going on to be able to assist.

[MoncefB]

Hello @carltongibson

Thank you for your prompt answer!

Yes, I understand.

I tried to python manage.py runserver 0.0.0.0:8000, and it worked.

It also worked with daphne with this simpler command: /home/ubuntu/Envs/charlie_env/bin/daphne -b 0.0.0.0 -p 8000 charlie_web_project.asgi:application when I'm hitting directly the server port 8000 from my locale machine.

It stops working when I start supervisor with those settings, if I try to directly hit port 8000.

But the most surprising is that it also stops working if I just use the simple Daphne command right above and proxy via nginx! Does it mean there is something wrong with my nginx config? Or my daphne settings to be able to use supervisor?

[carltongibson]

Good progress.

So what EXACTLY is Nginx sending? And why/how is that wrong?

Can you turn up the logging/set a breakpoint to see the request exactly as Daphe is receiving it. That would give you your answer I’d hope.

[MoncefB]

Thank you so much for your help, once again!

So when I hit directly the port, I get this in the Daphne logs: 2019-01-26 05:06:20,902 DEBUG connect <socket.socket fd=16, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('0.0.0.0', 0)> to ('172.31.1.185', 6379) 2019-01-26 05:06:20,903 DEBUG connect <socket.socket fd=17, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('0.0.0.0', 0)> to ('172.31.1.185', 6379) 2019-01-26 05:06:20,905 DEBUG poll 646.173 ms took 0.377 ms: 1 events 2019-01-26 05:06:20,906 DEBUG <socket.socket fd=16, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.31.23.28', 34452), raddr=('172.31.1.185', 6379)> connected to ip-172-31-1-185.ec2.internal:6379: (<_SelectorSocketTransport fd=16 read=polling write=<idle, bufsize=0>>, <asyncio.streams.StreamReaderProtocol object at 0x7fbe82cbb550>) 2019-01-26 05:06:20,909 DEBUG <socket.socket fd=17, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.31.23.28', 34454), raddr=('172.31.1.185', 6379)> connected to ip-172-31-1-185.ec2.internal:6379: (<_SelectorSocketTransport fd=17 read=polling write=<idle, bufsize=0>>, <asyncio.streams.StreamReaderProtocol object at 0x7fbe81aee080>) 2019-01-26 05:06:20,912 DEBUG poll 638.935 ms took 0.278 ms: 1 events 2019-01-26 05:06:20,914 DEBUG poll 637.504 ms took 1.234 ms: 1 events 2019-01-26 05:06:20,916 DEBUG WebSocket ['73.231.55.149', 51070] open and established 73.231.55.149:51070 - - [26/Jan/2019:05:06:20] "WSCONNECT /ws/v1.0/office/" - - 2019-01-26 05:06:20,916 DEBUG WebSocket ['73.231.55.149', 51070] accepted by application AKA all good

But when I hit the domain name, I get this instead: Not Found: /ws/v1.0/office/ 2019-01-26 05:12:31,731 WARNING Not Found: /ws/v1.0/office/ 2019-01-26 05:12:31,731 DEBUG poll 220.807 ms took 2.759 ms: 1 events 2019-01-26 05:12:31,732 DEBUG HTTP 404 response started for ['127.0.0.1', 59324] 2019-01-26 05:12:31,732 DEBUG poll 217.243 ms took 0.182 ms: 1 events 2019-01-26 05:12:31,733 DEBUG HTTP close for ['127.0.0.1', 59324] 2019-01-26 05:12:31,733 INFO "127.0.0.1" - - [03/Jan/1970:09:11:51 +0000] "GET /ws/v1.0/office/ HTTP/1.1" 404 17916 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1"

So either my Nginx config is wrong (it proxies everything as HTTP?), or my Daphne one (it should know something more about Nginx?) is wrong?

[MoncefB]

I finally understood where the issue was coming from, and fixed it!

Thank you @carltongibson for your help with the debugging!

And if anyone struggles the same way: it came from the AWS ELB, that doesn't support websockets on HTTP/HTTPS! But there is a work-around, and https://www.built.io/blog/websockets-on-aws-s-elb is a good resource to fix it!

Best,

[PradheepManimaran]

@carltongibson @MoncefB Hi, I am experiencing the same issue with Daphne and WebSockets. It works well in development, but after deploying to production on the Apache2 server, it throws a 404 error.

help me