ERR failed to connect to origin error="websocket: bad handshake"

baubukas commented 3 years ago

Server/Client: cloudflared version 2021.2.5 (built 2021-02-23-1951 UTC) (same issue with Windows and RPM clients) DNS records (Proxied) created through "cloudflared tunnel route dns" or running adhoc tunnel.

Error (with multiple subdomains): 2021-02-24T21:42:37Z ERR failed to connect to origin error="**websocket: bad handshake"** originURL=https://test.dblngtvnfnt.eu

With debug log level on client side getting interesting redirect to HTTP (Location: http://test.dblngtvnfnt.eu): 2021-02-24T21:42:37Z DBG Websocket response: "HTTP/1.1 301 Moved Permanently\r\nTransfer-Encoding: chunked\r\nCache-Control: max-age=3600\r\nCf-Ray: 626c5b80de6032aa-CDG\r\nCf-Request-Id: 0877978484000032aa01148000000001\r\nConnection: keep-alive\r\nDate: Wed, 24 Feb 2021 21:42:36 GMT\r\nExpect-Ct: max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"\r\nExpires: Wed, 24 Feb 2021 22:42:36 GMT\r\nLocation: http://test.dblngtvnfnt.eu/\r\nNel: {\"report_to\":\"cf-nel\",\"max_age\":604800}\r\nReport-To: {\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report?s=N1qzu5FBgOybuOx4tfcnVJd3jnq5hrCJTngvCwPCQNPwOz%2By8eB18i20FdmCVVE9rbCD3IXNs42g7PHCYF2fg%2Fg4zdBdbbnUt0BBxuVPZVD9cT5S\"}],\"max_age\":604800,\"group\":\"cf-nel\"}\r\nServer: cloudflare\r\n\r\n0\r\n\r\n"

nmldiegues commented 3 years ago

Hello @baubukas ,

Can you provide us with more details?

what is the origin cloudflared config
what are you doing on the client side, also using cloudflared for access?
is 2021.2.5 used both on origin as well as client side?

baubukas commented 3 years ago

Origin version:

RHEL 8.3 (rpm) cloudflared version 2021.2.5 (built 2021-02-23-1951 UTC)

Client version:

RHEL 8.3 (rpm) - cloudflared version 2021.2.5 (built 2021-02-23-1951 UTC)
Windows 10 (2004) - cloudflared version 2021.2.5 (built 2021-02-23-1951 UTC)

===================================================================== Origin config (adhoc):

cloudflared tunnel --hostname test.dblngtvnfnt.eu --url ssh://localhost:22

2021-02-24T22:52:22Z INF Connection established connIndex=0 location=ARN
2021-02-24T22:52:29Z INF Connection established connIndex=1 location=TXL
2021-02-24T22:52:30Z INF Connection established connIndex=2 location=ARN
2021-02-24T22:52:31Z INF Connection established connIndex=3 location=TXL
2021-02-24T22:52:52Z INF Each HA connection's tunnel IDs: map[0:5kyvnn7p4vl0zba4nits73up3n1ua6q4nn3tngfqted77zdzxghg 1:5kyvnn7p4vl0zba4nits73up3n1ua6q4nn3tngfqted77zdzxghg 2:5kyvnn7p4vl0zba4nits73up3n1ua6q4nn3tngfqted77zdzxghg 3:5kyvnn7p4vl0zba4nits73up3n1ua6q4nn3tngfqted77zdzxghg]
2021-02-24T22:52:52Z INF Route propagating, it may take up to 1 minute for your new route to become functional

===================================================================== Origin config (pre-configured):

/~/.cloudflare/config.yml:
    tunnel: test
    hostname: test.dblngtvnfnt.eu
    url: ssh://localhost:22
    credentials-file: /etc/cloudflared/<uuid>.json

ID                                   NAME        CREATED              CONNECTIONS  
<uuid>                               test 2021-02-24T17:50:13Z 2xDME, 2xTLL

===================================================================== Client side (in case of Windows):

cloudflared.exe access ssh --hostname test.dblngtvnfnt.eu --url localhost:22222

Then with putty.exe SSH to localhost:22222

===================================================================== Client side (in case of RHEL):

/root/.ssh/config:
Host test.dblngtvnfnt.eu
    ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h

Then simply ssh someuser@test.dblngtvnfnt.eu

=====================================================================

Always same result after authentication via Okta IdP:

Origin - preconfigured; Client - Windows 10 (2004)
Origin - adhoc; Client - Windows 10 (2004)
Origin - preconfigured; Client - RHEL 8.3 (rpm)
Origin - adhoc; Client - RHEL 8.3 (rpm)

nmldiegues commented 3 years ago

Thank you for the detailed report. This seems a pretty standard usage, of which we have several usages seemingly fine at the moment.

Can you confirm:

if this is deterministic?
if it only happens with this specific version?

nmldiegues commented 3 years ago

One thing that caught my attention:

when you run the "adhoc" version, you are not passing the tunnel name, meaning that it runs a non-named classic tunnel
when you run with the config, I suppose you call it with tunnel run, and it picks up the tunnel named test

in case 2, you are using the same hostname as case 1

However, case 1 is setting up DNS record to point to the classic tunnel, and so when you run case 2 without doing anything else, DNS will point to the tunnel of case 1 (which I guess you do not have running, as you are either running case 1 or 2).

The explanation is similar to what I gave today in https://community.cloudflare.com/t/how-can-i-run-argo-tunnel-with-config-yml/247326/3

baubukas commented 3 years ago

Just started playing with CF Access yesterday, so tried only this 2021.2.5 version. Later will try some earlier binaries on client and/or origin side and will try to compare outcome/debug logs.

nmldiegues commented 3 years ago

I just tried to reproduce your steps and I got the same error only when my origin had no SSH server listening on localhost:22. Otherwise it worked flawlessly.

nmldiegues commented 3 years ago

And in that case I observed this in the logs:

Origin/Server:

2021-02-24T23:29:56Z ERR Cannot connect to remote error="dial tcp [::1]:22: connect: connection refused"
2021-02-24T23:29:56Z ERR CF-RAY: 626cf8ba1c76da76-LIS Proxying to ingress 0 error: websocket: bad handshake

Eyeball/Client:

2021-02-24T23:29:56Z ERR failed to connect to origin error="websocket: bad handshake" originURL=<mytunnelhost-redacted>

baubukas commented 3 years ago

One thing that caught my attention:

when you run the "adhoc" version, you are not passing the tunnel name, meaning that it runs a non-named classic tunnel

when you run with the config, I suppose you call it with tunnel run, and it picks up the tunnel named test

in case 2, you are using the same hostname as case 1

However, case 1 is setting up DNS record to point to the classic tunnel, and so when you run case 2 without doing anything else, DNS will point to the tunnel of case 1 (which I guess you do not have running, as you are either running case 1 or 2).

The explanation is similar to what I gave today in https://community.cloudflare.com/t/how-can-i-run-argo-tunnel-with-config-yml/247326/3

Yes, I ran "adhoc" just to check maybe I was incorrectly defining - conf.yml; Yes it created AAAA record and is visible in Dashboard.
TBH, I am testing separate subdomains for "adhoc" and "pre-defined" tunnels on origin side - but same result.

baubukas commented 3 years ago

And in that case I observed this in the logs:

Origin/Server:

2021-02-24T23:29:56Z ERR Cannot connect to remote error="dial tcp [::1]:22: connect: connection refused"
2021-02-24T23:29:56Z ERR CF-RAY: 626cf8ba1c76da76-LIS Proxying to ingress 0 error: websocket: bad handshake

Eyeball/Client:

2021-02-24T23:29:56Z ERR failed to connect to origin error="websocket: bad handshake" originURL=<mytunnelhost-redacted>

# ping localhost
PING localhost(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from localhost (::1): icmp_seq=2 ttl=64 time=0.069 ms

# netstat -nao
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       Timer
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      off (0.00/0/0)
tcp        0    320 10.48.8.94:22           10.48.10.193:58916      ESTABLISHED on (0.22/0/0)
tcp6       0      0 :::22                   :::*                    LISTEN      off (0.00/0/0)

Of course SSHing to origin to configure it, but maybe I lack understanding how tunneling works. How to enable debug log level on origin side? Thanks.

nmldiegues commented 3 years ago

Logging on the origin side is the same as on the client side: --loglevel: debug after tunnel, or simply loglevel: debug in the config YAML.

morpig commented 3 years ago

experiencing the same issue too on my prev issue #309, and it looks like it only occurs on tcp connections (ssh/rdp/etc).

bangonkali commented 3 years ago

@nmldiegues , I have the same issue too, with the latest version. This only happens if I enable ACCESS POLICY. If there is no policy it correctly assumes the connection is SSH, but if there is an ACCESS POLICY then this error occurs:

➜ ssh 'devengr@gitlab-ssh.mydomain.com'
2021-02-26T20:39:15Z ERR failed to connect to origin error="websocket: bad handshake" originURL=https://gitlab-ssh.mydomain.com
websocket: bad handshake
kex_exchange_identification: Connection closed by remote host

If there is no ACCESS POLICY I can connect to SSH/Clone correctly.

nmldiegues commented 3 years ago

I have tried to reproduce that to no avail:

cloudflared 2021.2.5 in both origin and eyeball
SSH worked fine without Access Policy
added Access Policy, and it still works fine

This is a pretty simple PoC that maybe you can try to reproduce on your end:

Started an SSH server as origin (since my development laptop does not allow SSH) docker run -p 8222:2222 -e USER_PASSWORD=??? -e PASSWORD_ACCESS=true -e USER_NAME=??? ghcr.io/linuxserver/openssh-server

Tested direct SSH: ssh localhost -p 8222

Started origin tunnel: cloudflared tunnel --url ssh://localhost:8222 --hostname <Your Host>

Ran SSH over cloudflared access: ssh -oProxyCommand="cloudflared access ssh --hostname <Your Host>" -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null ???@<Your Host>

Then added the Access Policy, and the SSH over cloudflared access still worked (it prompted my browser for an access approval).

morpig commented 3 years ago

I think if you run it for less than an hour or minutes, it's still okay. Maybe try running the SSH connection over Argo for 24 hours.. or maybe an RDP connection. The problem occurs randomly, so it's hard to reproduce or estimate when will it happen..

This is what I've tried:

Use two different VMs in different locations, connected to different CF colo, all of them are dropped at the same time (websocket: bad handshake)
Running older Argo versions (last year builds), apparently it's the same. I remember not having any issues running it last year (except if edge is dropping/having issues)

Because of that, I thought of the problem being the tunnel-side issue. I checked open tcp connections, maximum ports, file descriptors, all seems to be ok.

nmldiegues commented 3 years ago

@morpig I see, that's a different type of issue, for which I recommend opening a ticket with our support for your Cloudflare account. If there's any problem, that'll take some digging specific to the traffic of your account, which we do not have knowledge of here.

@baubukas does anything on this thread help you? I'm afraid we're conflating different issues in the same thread and losing track of the original issue.

baubukas commented 3 years ago

@nmldiegues After all testing I suspect it is issue on CF Access side, but I have no idea how troubleshoot or how "working" logs should look. Strangest thing is that I do not see any indication that origin is receiving anything, even like it was in yours example

2021-02-24T23:29:56Z ERR Cannot connect to remote error="dial tcp [::1]:22: connect: connection refused"

nmldiegues commented 3 years ago

Are you saying that you also have an Access Policy @baubukas ? And that without it, it works?

baubukas commented 3 years ago

I was testing with policy only, will test setup without.

baubukas commented 3 years ago

Ok, so without Access Policy setup works for HTTP/S based services, however when TCP/SSH is attempted, on server side I do not get anything in logs (debug level), and on client side i get error="websocket: bad handshake"... In addition if service on orgin is defined as SSH and simply from browser I go to FQDN - on server side I get:

localhost:22 is a not http service

So it is not origin/tunnel side issue... it is more like client side with TCP/SSH.

nmldiegues commented 3 years ago

Ok, so without Access Policy setup works for HTTP/S based services, however when TCP/SSH is attempted, on server side I do not get anything in logs (debug level), and on client side i get error="websocket: bad handshake"...

Can you try reproducing the steps that I documented in https://github.com/cloudflare/cloudflared/issues/324#issuecomment-786945560 without Access Policy? It would be great if we could start from a common basis. Those should work.

baubukas commented 3 years ago

Can you try reproducing the steps that I documented in #324 (comment) without Access Policy? It would be great if we could start from a common basis. Those should work.

Docker:

[root@cloudlfared-0 tmp]# docker ps
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID  IMAGE                               COMMAND  CREATED         STATUS             PORTS                   NAMES
56bf6e70b0f5  ghcr.io/linuxserver/openssh-server           15 minutes ago  Up 15 minutes ago  0.0.0.0:8222->2222/tcp  clever_bartik

Local SSH:

[root@cloudlfared-0 tmp]# ssh user1@localhost -p 8222
The authenticity of host '[localhost]:8222 ([127.0.0.1]:8222)' can't be established.
ECDSA key fingerprint is SHA256:pggySCy58eVKCYyGRAkcmJjvLZ8vHQwfXB0z3PmrRJc.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:8222' (ECDSA) to the list of known hosts.
user1@localhost's password: 
Welcome to OpenSSH Server

56bf6e70b0f5:~$

Argo Tunnel:

[root@cloudlfared-0 tmp]# cloudflared tunnel --url ssh://localhost:8222 --hostname test.dblngtvnfnt.eu
2021-03-02T11:16:44Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
2021-03-02T11:16:44Z INF Version 2021.2.5
2021-03-02T11:16:44Z INF GOOS: linux, GOVersion: devel +11087322f8 Fri Nov 13 03:04:52 2020 +0100, GoArch: amd64
2021-03-02T11:16:44Z INF Settings: map[hostname:test.dblngtvnfnt.eu url:ssh://localhost:8222]
2021-03-02T11:16:44Z INF cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/argo-tunnel/reference/service/
2021-03-02T11:16:44Z INF Initial protocol h2mux
2021-03-02T11:16:44Z INF Starting metrics server on 127.0.0.1:34463/metrics
2021-03-02T11:16:44Z INF Connection established connIndex=0 location=ARN
2021-03-02T11:16:46Z INF Each HA connection's tunnel IDs: map[0:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg]
2021-03-02T11:16:46Z INF Route propagating, it may take up to 1 minute for your new route to become functional
2021-03-02T11:16:46Z INF Connection established connIndex=1 location=TXL
2021-03-02T11:16:47Z INF Connection established connIndex=2 location=ARN
2021-03-02T11:16:48Z INF Each HA connection's tunnel IDs: map[0:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg 1:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg]
2021-03-02T11:16:48Z INF Route propagating, it may take up to 1 minute for your new route to become functional
2021-03-02T11:16:48Z INF Connection established connIndex=3 location=TXL
2021-03-02T11:16:49Z INF Each HA connection's tunnel IDs: map[0:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg 1:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg 2:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg]
2021-03-02T11:16:49Z INF Route propagating, it may take up to 1 minute for your new route to become functional
2021-03-02T11:16:50Z INF Each HA connection's tunnel IDs: map[0:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg 1:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg 2:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg 3:vqf5zzicunsb5zyyz4nn20cxfga72raea4zgqey7b8ypsaahzalg]
2021-03-02T11:16:50Z INF Route propagating, it may take up to 1 minute for your new route to become functional

Backend side: CF-DNS CF-ArgoTunnel

Client side:

[root@cloudflared-1 tmp]# ssh -oProxyCommand="cloudflared access ssh --hostname test.dblngtvnfnt.eu" -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null osadmin@test.dblngtvnfnt.eu
2021-03-02T11:19:23Z ERR failed to connect to origin error="websocket: bad handshake" originURL=https://test.dblngtvnfnt.eu

nmldiegues commented 3 years ago

Can you run cloudflared access in the same box/container/vm as where you are running the origin tunnel, just to rule out networking issues?

baubukas commented 3 years ago

Can you run cloudflared access in the same box/container/vm as where you are running the origin tunnel, just to rule out networking issues?

Tried in two ways.

[root@cloudlfared-0 tmp]# cloudflared access ssh --hostname test.dblngtvnfnt.eu
2021-03-02T12:02:17Z ERR failed to connect to origin error="websocket: bad handshake" originURL=https://test.dblngtvnfnt.eu
websocket: bad handshake

[root@cloudlfared-0 tmp]# cloudflared access ssh --hostname test.dblngtvnfnt.eu --url ssh://localhost:22222
2021-03-02T12:05:49Z INF Start Websocket listener host=localhost:22222
2021-03-02T12:06:53Z ERR failed to connect to origin error="websocket: bad handshake" originURL=https://test.dblngtvnfnt.eu

Just in case SELinux / firewalld status:

[root@cloudlfared-0 tmp]# firewall-cmd --state
not running
[root@cloudlfared-0 tmp]# getenforce
Permissive

Just in case CA Trust:

[root@cloudlfared-0 tmp]# trust list
pkcs11:id=%85%30%5d%3b%2a%70%d4%ed%d5%92%67%07%fd%eb%39%b4%1a%0e%38%a7;type=cert
    type: certificate
    label: CloudFlare Origin SSL ECC Certificate Authority
    trust: anchor
    category: authority

pkcs11:id=%24%e8%53%57%5d%7c%34%40%87%a9%eb%94%db%ba%e1%16%78%fc%29%a4;type=cert
    type: certificate
    label: CloudFlare Origin SSL Certificate Authority
    trust: anchor
    category: authority

Very very odd overall, and still I sometimes get:

2021-03-02T11:16:50Z INF Route propagating, it may take up to 1 minute for your new route to become functional
2021-03-02T11:34:35Z ERR localhost:8222 is not a http service
**2021-03-02T11:34:35Z ERR CF-RAY: 629a511b2ab70bed-AMS Proxying to ingress 0 error: Not a http service**
2021-03-02T11:50:05Z INF Initiating graceful shutdown due to signal interrupt ...

Which shows that after all tunnel is working and as mentioned previously I have no issues with HTTP/S, but TCP/SSH is no go.

nmldiegues commented 3 years ago

Let's try to isolate this further into an even simpler example that's independent of your account: can you follow the same steps, but when running the origin tunnel drop --hostname <Your Host>. That will create a zoneless tunnel in Cloudflare trycloudflare.com domain.

For sanity check, I just re-did the steps with it and it also worked. Can you retry, to see if your problem is in any way related with your zone?

origin:

docker run -p 8222:2222 -e USER_PASSWORD=pass -e PASSWORD_ACCESS=true -e USER_NAME=user ghcr.io/linuxserver/openssh-server

tunnel:

cloudflared tunnel --url ssh://localhost:8222

access:

ssh -oProxyCommand="cloudflared access ssh --hostname <DOMAIN>.trycloudflare.com" -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null user@<DOMAIN>.trycloudflare.com

baubukas commented 3 years ago

For sanity check, I just re-did the steps with it and it also worked. Can you retry, to see if your problem is in any way related with your zone?

Container:

Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID  IMAGE                               COMMAND  CREATED        STATUS            PORTS                   NAMES
ceb37c12254a  ghcr.io/linuxserver/openssh-server           6 seconds ago  Up 6 seconds ago  0.0.0.0:8222->2222/tcp  hopeful_hofstadter

Local ssh to container:

[root@cloudlfared-0 tmp]# ssh user1@localhost -p 8222
The authenticity of host '[localhost]:8222 ([127.0.0.1]:8222)' can't be established.
ECDSA key fingerprint is SHA256:5/13oKzfjfmpp2y0JLOClt3fax1TuB0RDEK1vbJRdrU.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:8222' (ECDSA) to the list of known hosts.
user1@localhost's password: 
Welcome to OpenSSH Server

ceb37c12254a:~$

Argo tunnel:


[root@cloudlfared-0 tmp]# cloudflared tunnel --url ssh://localhost:8222
2021-03-02T12:34:31Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
2021-03-02T12:34:31Z INF Version 2021.2.5
2021-03-02T12:34:31Z INF GOOS: linux, GOVersion: devel +11087322f8 Fri Nov 13 03:04:52 2020 +0100, GoArch: amd64
2021-03-02T12:34:31Z INF Settings: map[url:ssh://localhost:8222]
2021-03-02T12:34:31Z INF cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/argo-tunnel/reference/service/
2021-03-02T12:34:31Z INF Initial protocol h2mux
2021-03-02T12:34:31Z INF Starting metrics server on 127.0.0.1:39697/metrics
2021-03-02T12:34:31Z INF Connection established connIndex=0 location=ARN
2021-03-02T12:34:33Z INF Each HA connection's tunnel IDs: map[0:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0]
2021-03-02T12:34:33Z INF +-------------------------------------------------------+
2021-03-02T12:34:33Z INF |  Your free tunnel has started! Visit it:              |
2021-03-02T12:34:33Z INF |    https://crop-bbs-nascar-passion.trycloudflare.com  |
2021-03-02T12:34:33Z INF +-------------------------------------------------------+
2021-03-02T12:34:33Z INF Route propagating, it may take up to 1 minute for your new route to become functional
2021-03-02T12:34:33Z INF Connection established connIndex=1 location=TXL
2021-03-02T12:34:34Z INF Connection established connIndex=2 location=ARN
2021-03-02T12:34:34Z INF Each HA connection's tunnel IDs: map[0:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0 1:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0]
2021-03-02T12:34:34Z INF +-------------------------------------------------------+
2021-03-02T12:34:34Z INF |  Your free tunnel has started! Visit it:              |
2021-03-02T12:34:34Z INF |    https://crop-bbs-nascar-passion.trycloudflare.com  |
2021-03-02T12:34:34Z INF +-------------------------------------------------------+
2021-03-02T12:34:34Z INF Route propagating, it may take up to 1 minute for your new route to become functional
2021-03-02T12:34:35Z INF Connection established connIndex=3 location=TXL
2021-03-02T12:34:35Z INF Each HA connection's tunnel IDs: map[0:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0 1:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0 2:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0]
2021-03-02T12:34:35Z INF +-------------------------------------------------------+
2021-03-02T12:34:35Z INF |  Your free tunnel has started! Visit it:              |
2021-03-02T12:34:35Z INF |    https://crop-bbs-nascar-passion.trycloudflare.com  |
2021-03-02T12:34:35Z INF +-------------------------------------------------------+
2021-03-02T12:34:35Z INF Route propagating, it may take up to 1 minute for your new route to become functional
2021-03-02T12:34:36Z INF Each HA connection's tunnel IDs: map[0:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0 1:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0 2:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0 3:79i9k4zcfs8scgqfyy03rpzyp673a3ucue34t1ut73rgt7uc7vq0]
2021-03-02T12:34:36Z INF +-------------------------------------------------------+
2021-03-02T12:34:36Z INF |  Your free tunnel has started! Visit it:              |
2021-03-02T12:34:36Z INF |    https://crop-bbs-nascar-passion.trycloudflare.com  |
2021-03-02T12:34:36Z INF +-------------------------------------------------------+
2021-03-02T12:34:36Z INF Route propagating, it may take up to 1 minute for your new route to become functional

Client side:

[root@cloudflared-1 tmp]# ssh -oProxyCommand="cloudflared access ssh --hostname crop-bbs-nascar-passion.trycloudflare.com" -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null user1@crop-bbs-nascar-passion.trycloudflare.com
2021-03-02T12:36:20Z ERR failed to connect to origin error="read tcp 10.48.8.84:53444->104.18.0.142:443: read: connection reset by peer" originURL=https://crop-bbs-nascar-passion.trycloudflare.com
read tcp 10.48.8.84:53444->104.18.0.142:443: read: connection reset by peer
kex_exchange_identification: Connection closed by remote host
[root@cloudflared-1 tmp]#

HTTP/S is attempted, seems that tunnel works:

 ERR localhost:8222 is not a http service
2021-03-02T12:37:22Z ERR CF-RAY: 629aad110e170820-TXL Proxying to ingress 0 error: Not a http service

To add more confusion it works from other client (Windows) now:

cloudflared.exe access ssh --hostname crop-bbs-nascar-passion.trycloudflare.com --url ssh://localhost:22222

login as: user1
user1@localhost's password:
Welcome to OpenSSH Server

ceb37c12254a:~$

and it works also from Linux client (however from other public IP). So:

trycloudflare.com - works overall
no issues with Windows client (Site 1)
no issues with Linux client (Site 2)
there are issues when client is within lab (Site 3), might be some security appliance in the path (connection reset by peer)

baubukas commented 3 years ago

Finally, found an issue... this definitely has to be added to documentation of Argo Tunnel/CF Access/CF for Teams - impact of SSL/TLS settings when Universal SSL is enabled/disabled.

On my test account Universal SSL was enabled, but SSL/TLS encryption mode was OFF. If Universal SSL is enabled, SSL/TLS encryption mode must be Flexible/Full/Full(strict) - it solves ""websocket: bad handshake"" issue for me.

@nmldiegues thanks for assistance.

adamchalmers commented 3 years ago

That's a great find, thank you.

@TownLake @abracchi-tw Do you know what the best place to document this is? I'm not sure where this should go, but we should document the resolution from the issue immediately above

adamchalmers commented 3 years ago

OK, it's being added to the docs and to our support handbook. Thanks all.

FR46M3N7-P4R71CL3 commented 3 years ago

Finally, found [the?] issue... this definitely has to be added to documentation of Argo Tunnel/CF Access/CF for Teams - impact of SSL/TLS settings when Universal SSL is enabled/disabled.

On my test account Universal SSL was enabled, but SSL/TLS encryption mode was OFF. If Universal SSL is enabled, SSL/TLS encryption mode must be Flexible/Full/Full(strict) - it solves ""websocket: bad handshake"" issue for me.

@nmldiegues thanks for assistance.

Unfortunately, not. I'm having Strict SSL (and tried others as well), running the same version of cloudflared on both client and the server and facing this issue... Let me know how I can I help to troubleshoot/isolate the issue. Thanks! Running a single tunnel, multiple ingress rules, with other rules working at the time when SSH from a native-client is failing.

Edit: My SSH domain is like: sub.domain.cc.cc.ext. Wondering if this could cause issues with cloudflared or with the service, with the below error:

newtack commented 3 years ago

I have the same issue with flexible ssl.

{"level":"error","error":"Not a http service","cfRay":"648c2dd47a7aef2e-MIA","ingressRule":"0","time":"2021-05-01T21:42:19Z"}

nighttardis commented 3 years ago

Hopefully this can help in figuring this out, I'm running into the same issue. If I use my custom domain I get the "not a http service" error when trying to forward SSH, but if I use the trycloudflare.com test domain, it works fine. In comparing the headers from the --loglevel debug output, I noticed the requests to my custom domain do not have the

Connection:[Upgrade] 
Upgrade:[websocket]

headers, but the requests to the trycloudflare.com test domain do. Now I'm not sure if this is a chicken and the egg issue where something isn't getting logged, but with those headers missing it does make sense why the agent on the server would think it should be an HTTP request not a websockets request.

I'm running the same cloudflared version on the the server (Debian 11, cloudflared version 2021.4.0 (built 2021-04-07-2048 UTC)) and the client (Manjaro, cloudflared version 2021.4.0 (built 2021-05-07T19:27:57+00:00)), using the same server for my testing and just switching between

cloudflared tunnel --url ssh://localhost:22 --hostname terminal.<customdomain>.com --loglevel debug and cloudflared tunnel --url ssh://localhost:22 --loglevel debug

7ojo commented 3 years ago

2021-05-23T19:17:00Z ERR failed to connect to origin error="remote error: tls: handshake failure" originURL=https://a.b.c.d
remote error: tls: handshake failure
kex_exchange_identification: Connection closed by remote host

Got similar problems and errors here. I found that if you have too many levels on your subdomain then SSH rules doesn't work correctly in cloudflared. Above you see errors when I had sub domain a.b.c.d which doesn't work, but if I change it to b.c.d it starts working.

m-torin commented 3 years ago

I was able to resolve the web sockets error but enabling web sockets under the Network section. Sigh.

nighttardis commented 3 years ago

@m-torin, thanks that was what it was. SSH proxy command is working, but seems like the "beta" web based ssh terminal still doesn't want to work, but that's another battle for another day.

a73x7 commented 3 years ago

I ran into the "remote error: tls: handshake failure" using the "a.b.domain.tld" pattern and it ended up being a bad certificate. Cloudflare gives you a free edge cert for "*.domain.tld" but that doesn't cover multi-level subdomains. After adding an Advanced Certificate for "*.b.domain.tld" it worked perfectly!

Maybe the DNS dashboard could show a warning for any records with proxy enabled that don't have a valid edge cert? I've had other unrelated issues caused by the same thing in the past and it's always very hard to debug.

eschulma commented 3 years ago

I too had this problem, and the solution was to turn off the "Enable Binding Cookie" in the Application settings in Teams. So much wasted time, please add this to the documentation. Credit for the solution goes here: (https://community.cloudflare.com/t/ssh-over-cloudflare-tunnel-always-results-in-failed-to-connect-to-origin-error-websocket-bad-handshake-unless-i-use-the-web-terminal/280924/4)

joaomamede commented 3 years ago

+1 with the same problem but. My linux machines as clients work, I can ssh,sftp,sshfs,etc.

My windows 10 machine as client have the same problem

2021-11-11T06:36:12Z ERR failed to connect to origin error="read tcp IPADRESSBLABAL:53966->IPADDRESBLEBLE:443: wsarecv: An existing connection was forcibly closed by the remote host." originURL=https://test.server.com

I tried with .\cloudflared.exe access ssh --hostname test.server.com ssh -o ProxyCommand=".\cloudflared.exe access ssh --hostname %h" root@test.server.com .\cloudflared.exe access ssh --hostname test.server.com --url ssh://localhost:22222

My teams "Enable Binding Cookie" is off.

mark-mybaggage commented 3 years ago

I had the same problem:

2021-11-09T12:02:09Z ERR failed to connect to origin error="websocket: bad handshake" originURL=

For me this happened after I set the 'Definitely automated' of 'Configure Super Bot Fight Mode' to Challenge (this is under Firewall/Bots on the Cloudflare dashboard). Changing the setting back to Allow resolved the issue.

As the description of this setting is 'Definitely automated traffic typically consists of bad bots. Select an action for this traffic.', I was surprised that it impacted Tunnels.

joaomamede commented 3 years ago

ppened after I set the 'Definitely automated' of 'Configure Super Bot Fight Mode' to Challenge

It's off in my case Bot Fight Mode is not on.

bsde1234 commented 2 years ago

when binary files is used and execute it from absolute path gives error. within folder it works

potatoqualitee commented 2 years ago

On my Windows client, I had to remove the host from known_hosts to get it working again.

cloudflare / cloudflared

ERR failed to connect to origin error="websocket: bad handshake" #324