[Bug]: 'Error: WebSocket close with status code 1006' behind nginx proxy

zirui commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

OS/Web Information

Web Browser: edge
Local OS: macos
Remote OS: ubuntu
Remote Architecture: x86
code-server --version: 4.6.0

Steps to Reproduce

proxy with nginx:

location /code/ {
  proxy_pass https://0.0.0.0:9997/;
  proxy_http_version 1.1;
  proxy_set_header Host $host;
  proxy_set_header Upgrade $http_upgrade;
  proxy_read_timeout 86400;
  proxy_set_header Connection $connection_upgrade;
  proxy_set_header Accept-Encoding gzip;
}

run code-server

code-server  --cert MyCertificate.crt --cert-key MyKey.key --port 9997

visit from url: ${myhost}/code/

Expected

Code server opens up to home screen.

Actual

error log:

Logs

Screenshot/Video

No response

Does this issue happen in VS Code or GitHub Codespaces?

[X] I cannot reproduce this in VS Code.
[X] I cannot reproduce this in GitHub Codespaces.

Are you accessing code-server over HTTPS?

[X] I am using HTTP.

Notes

Why does code-server create a websocket connection with port 80? And how can this issue be solved?

The network environment of the host machine: The port 80 has been disabled by the administrator of the host machine, only a few ports are open.

code-asher commented 1 year ago

It looks like it is using 80 because you are accessing the website over http. It might be better to let NGINX handle TLS, so remove --cert and --cert-key, change proxy_pass to http, give NGINX the certificate and key, then visit https://${myhost}/code/.

zirui commented 1 year ago

Thanks for your replay @code-asher I'm sorry I made a mistake, the error above occurred when I visited using HTTP not HTTPS. Then I tried the following based on your suggestion:

give NGINX the certificate and key

  ssl_certificate       MyCertificate.crt;
  ssl_certificate_key   MyKey.key ;

start code-server
```
code-server --log=debug --port 9997
```
visit using https:
```
https://${myhost}/code/
```
but it still went wrong. error logs:

code-asher commented 1 year ago

Is this a self-signed certificate? Some browsers (like Chrome, not sure about Edge though) will have trouble connecting web sockets with self-signed certificates. If you see something like a gray or green lock then you should be OK but if not you may need to either get a certificate signed by a CA or trust the certificate on your local machine (you can do this manually or install mkcert to do it for you).

You might also want to check NGINX's logs and code-server's output just in case it is an error on the server side rather than the browser.

zirui commented 1 year ago

Yes, it is a self-signed certificate generated by OpenSSL I checked the nginx's logs, but didn't find any clues. error.log: no related logs access.log:

10.10.6.9 - - [18/Feb/2023:16:58:06 +0800] "GET /code/stable-2062a59ca1a586d8a6e7bf483841085a94c440a4/static/extensions/git-base/dist/browser/extension.js HTTP/1.0" 404 29 "https://xxx.yy/code/stable-2062a59ca1a586d8a6e7bf483841085a94c440a4/static/out/vs/base/worker/workerMain.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.41"
10.10.6.9 - - [18/Feb/2023:16:58:06 +0800] "GET /code/stable-2062a59ca1a586d8a6e7bf483841085a94c440a4/static/extensions/emmet/dist/browser/emmetBrowserMain.js HTTP/1.0" 404 29 "https://xxx.yy/code/stable-2062a59ca1a586d8a6e7bf483841085a94c440a4/static/out/vs/base/worker/workerMain.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.41"

zirui commented 1 year ago

I have tried the following two methods:

create locally signed SSL certificates with mkcert
access through safari browser

but neither of them worked.

Is it because the 443 port of the host machine is not open? Is there an alternative solution? My machine has very strict network restrictions, and only a few ports are open.

the output logs of the Edge: err.log

code-asher commented 1 year ago

Sorry for the delayed response.

You will need port 443 open on the machine running NGINX.

But you said you access via https://${myhost}/code/ right? That means you are already accessing through port 443 so it should already be open. Both https:// and wss:// default to port 443.

Interesting that there are no error logs from NGINX, that does suggest it might still be a certificate issue. Did you update NGINX to use the certificates generated by mkcert?

zirui commented 1 year ago

Sorry for the delayed response. You will need port 443 open on the machine running NGINX. But you said you access via https://${myhost}/code/ right? That means you are already accessing through port 443 so it should already be open. Both https:// and wss:// default to port 443. Interesting that there are no error logs from NGINX, that does suggest it might still be a certificate issue. Did you update NGINX to use the certificates generated by mkcert?

Hi, After some test on websocket connections, I found the problem: The security gateway on the host machine does not enable WebSocket when forwarding domain name, only HTTP/HTTPS requests will be forwarded. Therefore, there are always WebSocket connection errors in the browser front-end: [initial[xxx:443] socketFactory.connect() failed or timed out"

Due to security concerns, the administrator of the host machine will not allow WebSocket forwarding requests. Are there any alternative solutions?

code-asher commented 1 year ago

Ahh that makes sense!

There is currently no way to use code-server without web sockets so you would need to figure out some other way to get the web sockets to connect.

For example if you are able to use SSH you could access code-server through an SSH tunnel.

ssh -N -L 8080:127.0.0.1:8080 [user]@<instance-ip>

Then visit localhost:8080.

zirui commented 1 year ago

Ahh that makes sense! There is currently no way to use code-server without web sockets so you would need to figure out some other way to get the web sockets to connect. For example if you are able to use SSH you could access code-server through an SSH tunnel. ssh -N -L 8080:127.0.0.1:8080 [user]@<instance-ip> Then visit localhost:8080.

Unfortunately, the direct SSH connection to the remote host has also been blocked by the administrator (requiring connection through a jump host and do some authentication), so SSH tunnel won't work. Anyway, thank you for your reply.

SimonTod commented 1 year ago

I can confirm that this issue is present since 4.10.1. 4.10.0 and below work fine.

code-asher commented 1 year ago

@SimonTod 4.10.1 added a security check that uses the host and origin headers; do you have code-server behind a reverse proxy and does it forward the host header?

TomBraun02 commented 1 year ago

In my installation with Nginx-Proxy the following works with 4.11.0

location / { proxy_pass http://127.0.0.1:3443/; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection upgrade; proxy_set_header Accept-Encoding gzip; proxy_set_header Origin https://$host; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; }

docker run -it --name code-server -p 127.0.0.1:3443:8080 -v "$HOME/.config:/home/coder/.config" -v "$PWD:/home/coder/project" -u "$(id -u):$(id -g)" -e "DOCKER_USER=$USER" codercom/code-server:latest

code-asher commented 1 year ago

That makes sense but do keep in mind that if you are always setting the origin to the host then you are bypassing the security check.

However the exploit only happens on older browsers that do not support SameSite cookies and attacks across sub-domains that share the same root domain so if neither of these apply to you then there should be no problem.

maxorlovsky commented 1 year ago

@TomBraun02 solution worked for me, even though if it's not secure I would like to find a way to not bypass the check

code-asher commented 1 year ago

Just released 4.12.0 with some debug logs, running with --log debug should show why the requests are being blocked.

code-asher commented 1 year ago

Might need to use $http_host instead: https://github.com/coder/code-server/issues/6166

SimonTod commented 1 year ago

@code-asher Thanks a lot.

I had this next line missing in my reverse proxy config

proxy_set_header Host $host;

and this next one had to be commented

# proxy_set_header X-Forwarded-Host $remote_addr;

code-asher commented 1 year ago

Ahh, that makes sense, remote_addr is the IP address I think.

HeveraletLaidCenx commented 1 year ago

I seem to be having a similar problem, at least in terms of the error shown( 1006 and WS 403 from the network tab), but I'm using Caddy v2 as a reverse proxy, my network topology may not be best practice (as shown in the diagram), but I hope it is appropriate to post here.

To debug, I run code-server with the flag --log debug, and I found that when I access from WAN, the log report something like:

code-server 4.13.0 2798322b03e7f446f59c5142215c11711ed7a427
Using user-data-dir ~/.local/share/code-server
Using config file ~/.config/code-server/config.yaml
HTTP server listening on http://127.0.0.1:8080/
    - Authentication is enabled
      - Using password from ~/.config/code-server/config.yaml
    - Not serving HTTPS

Extension host agent started.
redirecting from / to ./login
No uninstalled extensions found.
debug 0 active connections
got cookie domain {"host":"MyPublicDomain.com"}                                              
redirecting from /login to ./
host "192.168.LAN.CaddyHost" does not match origin "MyPublicDomain.com"; blocking request to /stable-b***5?reconnectionToken=a***a&reconnection=false&skipWebSocketFrames=false
Forbidden HttpError: Forbidden
    at ensureOrigin (/usr/lib/code-server/out/node/http.js:295:15)
    at wrapped (/usr/lib/code-server/out/node/wsRouter.js:64:24)
    at Layer.handle [as handle_request] (/usr/lib/code-server/node_modules/router/lib/layer.js:102:15)
    at next (/usr/lib/code-server/node_modules/router/lib/route.js:144:13)
    at Route.dispatch (/usr/lib/code-server/node_modules/router/lib/route.js:109:3)
    at handle (/usr/lib/code-server/node_modules/router/index.js:515:11)
    at Layer.handle [as handle_request] (/usr/lib/code-server/node_modules/router/lib/layer.js:102:15)
    at /usr/lib/code-server/node_modules/router/index.js:291:22
    at param (/usr/lib/code-server/node_modules/router/index.js:368:14)
    at param (/usr/lib/code-server/node_modules/router/index.js:379:14)
    at Function.process_params (/usr/lib/code-server/node_modules/router/index.js:424:3)
    at next (/usr/lib/code-server/node_modules/router/index.js:285:10)
    at Function.handle (/usr/lib/code-server/node_modules/router/index.js:184:3)
    at router (/usr/lib/code-server/node_modules/router/index.js:59:12)
    at Layer.handle [as handle_request] (/usr/lib/code-server/node_modules/router/lib/layer.js:102:15)
    at trim_prefix (/usr/lib/code-server/node_modules/router/index.js:330:13)
    at /usr/lib/code-server/node_modules/router/index.js:294:7
    at Function.process_params (/usr/lib/code-server/node_modules/router/index.js:349:12)
    at Immediate.next (/usr/lib/code-server/node_modules/router/index.js:285:10)
    at Immediate.<anonymous> (/usr/lib/code-server/node_modules/router/index.js:671:15)
    at processImmediate (node:internal/timers:466:21)

... many errors like the last 2 above loop

I check the log of my LAN caddy server, it shows something I think may be the problem:

request>headers>Origin: "https://MyPublicDomain.com"
request>headers>X-Forwarded-Host: "MyPublicDomain.com"
request>host: "192.168.LAN.CaddyHost"

I tried to find a solution in the doc of caddy and code-server's FAQ and issues, From my understanding of the problem, it seems that setting a trust proxy on the backend is the solution, but it looks like that was deprecated in code-server many versions ago. I'm not sure if I should continue to tweak the caddy configuration or if there's any way to fix this, if there is no way at all that such a network topology will work?

code-asher commented 1 year ago

I think you will have to set trusted_proxies in the LAN Caddy so it trusts the X-Forwarded-* headers from the WAN Caddy and passes them along rather than overriding them.

https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#defaults

HeveraletLaidCenx commented 1 year ago

I think you will have to set trusted_proxies in the LAN Caddy so it trusts the X-Forwarded-* headers from the WAN Caddy and passes them along rather than overriding them. https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#defaults

Thanks for your help! After a few more debugging and testing, it finally worked, that's really saved my day! Perhaps I should note some additional info here if it can help someone stuck in a similar situation:

Keep the network topology AS SIMPLE AS YOU CAN to avoid hard-to-locate errors. Or, if you have to set a relatively complex cases that are not directly connected to code-server using one layer Caddy's reverse_proxy, just like mine, then, switch on all debug logs available during the whole route to locate the main issue. For me, some lack of network knowledge I guess led me to not accurately understand the default behavior of how caddy deal with requests.

Besides the trusted_proxies asher mentioned above, I found:

after setting it, the Host in my LAN caddy's log changed to the WAN Router's WAN IP:port, which lead to a blank response even did not reach code-server at all.

To fix that, I edit the Caddyfile of my WAN caddy, replace the header_up Host {upstream_hostport} to header_up Host 192.168.LAN.CaddyHost:port.

After reloading, everything worked. Remember to switch off those debug to set a production state! Hope this helped XD.

code-asher commented 1 year ago

Whoops, ignore my last message if you saw it. I replied to the wrong email.

Glad you got it working!

Izooc commented 7 months ago

In my installation with Nginx-Proxy the following works with 4.11.0

location / { proxy_pass http://127.0.0.1:3443/; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection upgrade; proxy_set_header Accept-Encoding gzip; proxy_set_header Origin https://$host; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; }

docker run -it --name code-server -p 127.0.0.1:3443:8080 -v "$HOME/.config:/home/coder/.config" -v "$PWD:/home/coder/project" -u "$(id -u):$(id -g)" -e "DOCKER_USER=$USER" codercom/code-server:latest

This works for me (yay) but I was wondering if there is a way to do it securely? As someone in this thread mentioned it could pose a security risk. Or am I just being thick... Thanks for this tho 🙏

(edit) I found the config for nginx on the docs and am using that, should be okay i think??

code-asher commented 7 months ago

Yeah the config in the docs is good. You want to avoid setting Origin in the proxy config, to prevent some cases of cross-site web socket hijacking. https://github.com/coder/code-server/blob/3e8100b70eedf1cd5d0c81c5e7cc35a55c0acc18/CHANGELOG.md#L195-L197

izayoi-akira commented 6 months ago

In my installation with Nginx-Proxy the following works with 4.11.0 location / { proxy_pass http://127.0.0.1:3443/; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection upgrade; proxy_set_header Accept-Encoding gzip; proxy_set_header Origin https://$host; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; } docker run -it --name code-server -p 127.0.0.1:3443:8080 -v "$HOME/.config:/home/coder/.config" -v "$PWD:/home/coder/project" -u "$(id -u):$(id -g)" -e "DOCKER_USER=$USER" codercom/code-server:latest

This works for me (yay) but I was wondering if there is a way to do it securely? As someone in this thread mentioned it could pose a security risk. Or am I just being thick... Thanks for this tho 🙏

(edit) I found the config for nginx on the docs and am using that, should be okay i think??

Is there the docs link?🙏

code-asher commented 6 months ago

https://github.com/coder/code-server/blob/3e8100b70eedf1cd5d0c81c5e7cc35a55c0acc18/docs/guide.md#using-lets-encrypt-with-nginx

coder / code-server