--openvpn does not live together with --ssh option

robonuka commented 9 months ago

Hello Yves! If I specify options: --listen=0.0.0.0:80 --openvpn=0.0.0.0:1194 --ssh=0.0.0.0:22 And then trying to connect with OpenVPN client to 80 port - SSLH always redirects me to the SSH 22 port. I tried to change places --openvpn and --ssh options places but no luck.( Only way to solve this problem is to run SSLH without --ssh option. Yves, fix this bug, please!

PS: Executing with user 'sslh': sslh -V sslh-select head-2024-02-03

yrutschle commented 9 months ago

Can you post sslh' s logs with some more verbose, in particular with verbose-probe-info enabled?

robonuka commented 9 months ago

SSLH version is:

Executing with user 'sslh': sslh --version
sslh-select head-2024-02-03

x.x.x.x - my PC public IP address z.z.z.z - SSLH server PC public IP address

Executing with user 'sslh': sslh --foreground --numeric --verbose-probe-info=7 --listen=0.0.0.0:80 --http=0.0.0.0:8080 --openvpn=0.0.0.0:9443 --ssh=0.0.0.0:122
ssh:connection from x.x.x.x:54524 to z.z.z.z:80 forwarded from 127.0.0.1:39414 to 127.0.0.1:122
ssh:connection from x.x.x.x:46826 to z.z.z.z:80 forwarded from 127.0.0.1:51326 to 127.0.0.1:122
ssh:connection from x.x.x.x:54036 to z.z.z.z:80 forwarded from 127.0.0.1:39074 to 127.0.0.1:122

OpenVPN client obviously can't connect with SSH server, but it trying)

If I remove --ssh=0.0.0.0:122 parameter then everything becomes fine!

Executing with user 'sslh': sslh --foreground --numeric --verbose-probe-info=7 --listen=0.0.0.0:80 --http=0.0.0.0:8080 --openvpn=0.0.0.0:9443
openvpn:connection from x.x.x.x:44798 to z.z.z.z:80 forwarded from 127.0.0.1:54624 to 127.0.0.1:9443

I have OpenVPN client under Ubuntu 22.04

openvpn --version
OpenVPN 2.5.9 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Sep 29 2023
library versions: OpenSSL 3.0.2 15 Mar 2022, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2022 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=no enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_maintainer_mode=no enable_management=yes enable_multihome=yes enable_option_checking=no enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_openssl_engine=yes with_sysroot=no

And OpenVPN Server version 2.6.8. (https://hub.docker.com/r/robonuka/openvpn)

yrutschle commented 9 months ago

Sorry, I meant verbose_packets... to have the hex dump of incoming packets on which probing is performed. What you are seeing suggests the OpenVPN probe isn' t working for you.

robonuka commented 9 months ago

I put --verbose-packets parameter but it does not help: I tried n = 3,5,7 but all tries have the same result. See screenshot. Without --ssh parameter also no results from verbose

robonuka commented 9 months ago

Wow! I see a hexdump! Unfortunately not for openvpn, but for some http requests

root@robonuka:/opt# docker logs -f sslh
Executing with user 'sslh': sslh --foreground --numeric --verbose-packets=7 --listen=0.0.0.0:80 --http=0.0.0.0:8080 --openvpn=0.0.0.0:9443
openvpn:connection from x.x.x.x:39952 to z.z.z.z:80 forwarded from 127.0.0.1:35632 to 127.0.0.1:9443
hexdump of incoming packet:
0x000000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
0x000010: 48 6f 73 74 3a 20 zz zz zz zz zz zz zz zz zz zz Host: zzz.zzz.zzz.
0x000020: zz zz 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 zzz..User-Agent: 
0x000030: 4d 6f 7a 69 6c 6c 61 2f 35 2e 30 20 28 57 69 6e Mozilla/5.0 (Win
0x000040: 64 6f 77 73 20 4e 54 20 36 2e 32 3b 65 6e 2d 55 dows NT 6.2;en-U
0x000050: 53 29 20 41 70 70 6c 65 57 65 62 4b 69 74 2f 35 S) AppleWebKit/5
0x000060: 33 37 2e 33 32 2e 33 36 20 28 4b 48 54 4d 4c 2c 37.32.36 (KHTML,
0x000070: 20 6c 69 76 65 20 47 65 63 6b 6f 29 20 43 68 72  live Gecko) Chr
0x000080: 6f 6d 65 2f 35 32 2e 30 2e 33 30 39 37 2e 37 31 ome/52.0.3097.71
0x000090: 20 53 61 66 61 72 69 2f 35 33 37 2e 33 32 0d 0a  Safari/537.32..
0x0000a0: 41 63 63 65 70 74 2d 45 6e 63 6f 64 69 6e 67 3a Accept-Encoding:
0x0000b0: 20 67 7a 69 70 2c 20 64 65 66 6c 61 74 65 0d 0a  gzip, deflate..
0x0000c0: 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 43 6f 6e Accept: */*..Con
0x0000d0: 6e 65 63 74 69 6f 6e 3a 20 6b 65 65 70 2d 61 6c nection: keep-al
0x0000e0: 69 76 65 0d 0a 0d 0a                            ive....         
http:connection from 54.173.197.66:46466 to z.z.z.z:80 forwarded from 127.0.0.1:46704 to 127.0.0.1:8080
hexdump of incoming packet:
0x000000: 16 03 01 02 00 01 00 01 fc 03 03 38 e3 08 ab 44 ...........8...D
0x000010: 83 f9 ef f1 e9 fb 4d 8d de 9a dd 5a 5f 84 7c 79 ......M....Z_.|y
0x000020: 58 2f 1a d6 2c 04 2f 65 45 56 a7 20 7e a5 4c 29 X/..,./eEV. ~.L)
0x000030: 0f 93 84 3c e2 a8 db fd a9 1d 92 d6 ec e1 81 9b ...<............
0x000040: 83 74 8d 92 d1 2f 85 13 c9 65 af d5 00 56 13 02 .t.../...e...V..
0x000050: 13 03 13 01 c0 2c c0 30 c0 2b c0 2f cc a9 cc a8 .....,.0.+./....
0x000060: 00 9f 00 9e cc aa c0 af c0 ad c0 ae c0 ac c0 24 ...............$
0x000070: c0 28 c0 23 c0 27 c0 0a c0 14 c0 09 c0 13 c0 a3 .(.#.'..........
0x000080: c0 9f c0 a2 c0 9e 00 6b 00 67 00 39 00 33 00 9d .......k.g.9.3..
0x000090: 00 9c c0 a1 c0 9d c0 a0 c0 9c 00 3d 00 3c 00 35 ...........=.<.5
0x0000a0: 00 2f 00 ff 01 00 01 5d 00 0b 00 04 03 00 01 02 ./.....]........
0x0000b0: 00 0a 00 0c 00 0a 00 1d 00 17 00 1e 00 19 00 18 ................
0x0000c0: 00 10 00 0b 00 09 08 68 74 74 70 2f 31 2e 31 00 .......http/1.1.
0x0000d0: 16 00 00 00 17 00 00 00 31 00 00 00 0d 00 2a 00 ........1.....*.
0x0000e0: 28 04 03 05 03 06 03 08 07 08 08 08 09 08 0a 08 (...............
0x0000f0: 0b 08 04 08 05 08 06 04 01 05 01 06 01 03 03 03 ................
0x000100: 01 03 02 04 02 05 02 06 02 00 2b 00 05 04 03 04 ..........+.....
0x000110: 03 03 00 2d 00 02 01 01 00 33 00 26 00 24 00 1d ...-.....3.&.$..
0x000120: 00 20 c3 9f e2 84 e4 06 a1 6e ad 3c 32 ab b9 f4 . .......n.<2...
0x000130: df c1 a8 ed d6 0c 9c 27 38 65 e9 9e 22 4e 65 21 .......'8e.."Ne!
0x000140: 9a 19 00 15 00 bf 00 00 00 00 00 00 00 00 00 00 ................
0x000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x000190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0001b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0001d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x000200: 00 00 00 00 00                                  .....           
http:connection from 54.173.197.66:40458 to z.z.z.z:80 forwarded from 127.0.0.1:53884 to 127.0.0.1:8080
hexdump of incoming packet:
0x000000: 0d 0a                                           ..              
http:connection from 54.173.197.66:51540 to z.z.z.z:80 forwarded from 127.0.0.1:32828 to 127.0.0.1:8080
hexdump of incoming packet:
0x000000: 53 53 48 2d 32 2e 30 2d 6c 69 62 73 73 68 32 5f SSH-2.0-libssh2_
0x000010: 31 2e 31 31 2e 30 0d 0a                         1.11.0..

yrutschle commented 9 months ago

that's rather odd, as the hex dump occurs before any decision on protocols is made (i.e. the behaviour is the same for all incoming connections at that point). Are you sure you don't have half-dead sessions with previous configurations?

scmanjarrez commented 7 months ago

Hi, recently I have come across this problem, I'm trying to debug the connection but it seems that flags are not being detected.

/ # sslh -V
sslh-select head-2024-03-28
/ # sslh --foreground --numeric --verbose-packets=1 --verbose-probe-info=1 --verbose-connections=1 --verbose-connections-try=1 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:2
2
trying to connect to 127.0.0.1:22 family 2 len 16
ssh:connection from x.x.x.x:52514 to 10.0.0.226:443 forwarded from 127.0.0.1:55178 to 127.0.0.1:22
trying to connect to 127.0.0.1:22 family 2 len 16
ssh:connection from x.x.x.x:52515 to 10.0.0.226:443 forwarded from 127.0.0.1:51670 to 127.0.0.1:22
trying to connect to 127.0.0.1:22 family 2 len 16
ssh:connection from x.x.x.x:52516 to 10.0.0.226:443 forwarded from 127.0.0.1:47656 to 127.0.0.1:22

I've tried using --verbose-packets=7 (same as the OP), but same output. I've tried using --transparent mode as well, same.

I'm using a docker container, but I've changed the entrypoint to start sslh manually.

    entrypoint: sh
    command: -c 'tail -f /dev/null'

I've commented in this issue because I'm having the same problem, openvpn is redirected through ssh

yrutschle commented 7 months ago

On Thu, Mar 28, 2024 at 04:45:03AM -0700, Sergio C wrote:

/ # sslh -V sslh-select head-2024-03-28

This format is odd. Do you get this from a version checked out of the current Github head?

/ # sslh --foreground --numeric --verbose-packets=1 --verbose-probe-info=1 --verbose-connections=1 --verbose-connections-try=1 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:22 trying to connect to 127.0.0.1:22 family 2 len 16 ssh:connection from x.x.x.x:52514 to 10.0.0.226:443 forwarded from 127.0.0.1:55178 to 127.0.0.1:22 trying to connect to 127.0.0.1:22 family 2 len 16 ssh:connection from x.x.x.x:52515 to 10.0.0.226:443 forwarded from 127.0.0.1:51670 to 127.0.0.1:22 trying to connect to 127.0.0.1:22 family 2 len 16 ssh:connection from x.x.x.x:52516 to 10.0.0.226:443 forwarded from 127.0.0.1:47656 to 127.0.0.1:22
I've tried using --verbose-packets=7 (same as the OP), but same output.

I can't explain why you don't get the incoming packet dumps. verbose 1 outputs to stdout, while verbose 3 (or 7) output to stdout AND syslog, so maybe you can try to see if you get something in logs (it's logged under AUTH facility by default, which ends up in /var/log/auth.log in Debian).

I have no idea if docker would interfere with log emission...

scmanjarrez commented 7 months ago

Sorry, when I started using sslh the containers weren't provided. I was building my own containers with docker compose build definition build: https://github.com/yrutschle/sslh.

I've updated my compose file to provided containers.

/ # sslh -V
sslh-select head-2024-03-24
/ # ls /var/log/
/ #

I've tested different ways to run sslh, it worked once but I haven't been able to replicate it.

While doing the tests again, I was able to make it work once, but in another try, it didn't work, I'm glad this happened so I can show you both cases

root user:

/ # sslh --foreground --numeric --verbose-packets=7 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:22
ssh:connection from x.x.x.x:57379 to 10.0.0.226:443 forwarded from 127.0.0.1:37316 to 127.0.0.1:22

# another terminal
/ # ls /var/log/
/ #

sslh user (this time worked):

/ # su sslh
/ $ sslh --foreground --numeric --verbose-packets=7 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:22
hexdump of incoming packet:                                                                                                                                                                                 [10/90]
0x000000: 00 56 38 61 03 df d7 1d bc 8e 89 c2 e1 98 e3 92 .V8a............
0x000010: 90 4c 85 72 91 79 10 8b 27 f5 04 db 07 0b 12 4c .L.r.y..'......L
0x000020: 44 6b 6a d6 7e e2 b9 63 60 54 26 5c f9 38 9e 2f Dkj.~..c`T&\.8./
0x000030: 58 34 8d a6 39 13 1d bb 75 bb c4 22 31 ba 86 e6 X4..9...u.."1...
0x000040: fa dd ba 80 8f c5 35 78 1c 2c e7 00 00 00 01 66 ......5x.,.....f
0x000050: 05 82 e7 00 00 00 00 00 00 56 38 61 03 df d7 1d .........V8a....
0x000060: bc 8e 89 9a 6d 95 51 52 ab df 90 70 d2 dd 11 c2 ....m.QR...p....
0x000070: 00 b4 80 dc 4f c4 1f 51 79 8a 5a 90 90 93 d2 8c ....O..Qy.Z.....
0x000080: c7 35 58 a6 d3 69 ac 24 d0 5d a3 ce 7c d4 3b 3d .5X..i.$.]..|.;=
0x000090: e1 d6 a1 04 cb 53 3b 6c 7c 44 c3 b3 d2 8c f3 3a .....S;l|D.....:
0x0000a0: f1 c1 b0 00 00 00 02 66 05 82 e7 00 00 00 00 00 .......f........
openvpn:connection from x.x.x.x:57409 to 10.0.0.226:443 forwarded from 127.0.0.1:59830 to 127.0.0.1:4443
hexdump of incoming packet:
0x000000: 16 03 01 00 85 01 00 00 81 03 03 2b f7 78 65 23 ...........+.xe#
0x000010: f1 75 04 0d 0f 4f 21 fa 5a 03 4e bf dd f6 02 e5 .u...O!.Z.N.....
0x000020: 6c 2d b2 ac cc 1f de 51 d1 62 19 00 00 20 c0 2f l-.....Q.b... ./
0x000030: c0 30 c0 2b c0 2c cc a8 cc a9 c0 13 c0 09 c0 14 .0.+.,..........
0x000040: c0 0a 00 9c 00 9d 00 2f 00 35 c0 12 00 0a 01 00 ......./.5......
0x000050: 00 38 00 05 00 05 01 00 00 00 00 00 0a 00 0a 00 .8..............
0x000060: 08 00 1d 00 17 00 18 00 19 00 0b 00 02 01 00 00 ................
0x000070: 0d 00 0e 00 0c 04 01 04 03 05 01 05 03 02 01 02 ................
0x000080: 03 ff 01 00 01 00 00 12 00 00                   ..........
tls:connection from x.x.x.x:33420 to 10.0.0.226:443 forwarded from 127.0.0.1:40086 to 127.0.0.1:9443

# another terminal
/ # ls /var/log/
/ #

sslh user (this time didn't work, same configuration):

/ # su sslh
/ $ sslh --foreground --numeric --verbose-packets=7 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:22
ssh:connection from x.x.x.x:57512 to 10.0.0.226:443 forwarded from 127.0.0.1:51396 to 127.0.0.1:22
ssh:connection from x.x.x.x:57516 to 10.0.0.226:443 forwarded from 127.0.0.1:51408 to 127.0.0.1:22

# another terminal
/ # ls /var/log/
/ #

sslh user (invoking su with '-'):

/ # su - sslh
vps1:~$ sslh --foreground --numeric --verbose-packets=7 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:22
ssh:connection from x.x.x.x:57489 to 10.0.0.226:443 forwarded from 127.0.0.1:58138 to 127.0.0.1:22
ssh:connection from x.x.x.x:57492 to 10.0.0.226:443 forwarded from 127.0.0.1:58152 to 127.0.0.1:22

# another terminal
/ # ls /var/log/
/ #

I wasn't able to gather logs from /var/log in any case, it was always empty.

yrutschle commented 7 months ago

On Thu, Mar 28, 2024 at 07:54:20AM -0700, Sergio C wrote:

sslh user (this time didn't work, same configuration):

I'm thinking it might end up connecting to ssh because it's the last one, and the probing times out. Can you add --verbose-fd=1 ? This should give more insights one the file descriptor management, including connections that happen on timeout.

scmanjarrez commented 7 months ago

Yes, it's timing out, but should it at least show the incoming packets (as declared with --verbose-packets)? It's not taking into account that flag.

root user:


/ # sslh --foreground --numeric --verbose-fd=1 --verbose-packets=7 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:22
selecting... max_fd=4 num_probing=0
accepting from 3
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
...
...
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
timeout slot 0
timed out, connect to ssh
closing fd 4
selecting... max_fd=4 num_probing=0
ssh:connection from x.x.x.x:58057 to 10.0.0.226:443 forwarded from 127.0.0.1:33472 to 127.0.0.1:22
socket closed
connection closed down
accepting from 3

sslh user (tried twice, didn't work):

/ $ sslh --foreground --numeric --verbose-fd=1 --verbose-packets=7 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:22
selecting... max_fd=4 num_probing=0
accepting from 3
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
...
...
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
timeout slot 0
timed out, connect to ssh
closing fd 4
selecting... max_fd=4 num_probing=0
ssh:connection from x.x.x.x:58098 to 10.0.0.226:443 forwarded from 127.0.0.1:42306 to 127.0.0.1:22
socket closed
connection closed down

sslh user (dash):

/ $ sslh --foreground --numeric --verbose-fd=1 --verbose-packets=7 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:22
selecting... max_fd=4 num_probing=0
accepting from 3
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
...
...
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
timeout slot 0
timed out, connect to ssh
closing fd 4
selecting... max_fd=4 num_probing=0
ssh:connection from x.x.x.x:58127 to 10.0.0.226:443 forwarded from 127.0.0.1:60390 to 127.0.0.1:22
socket closed
connection closed down

I've increased the timeout to 10 seconds, but still sent to ssh

sslh --foreground --numeric --verbose-fd=1 --verbose-packets=7 --listen=0.0.0.0:443 --tls=localhost:9443 --openvpn=localhost:4443 --ssh=localhost:22 --timeout=10
selecting... max_fd=4 num_probing=0
accepting from 3
selecting... max_fd=4 num_probing=1
selecting... max_fd=4 num_probing=1
...
...
selecting... max_fd=4 num_probing=1
timeout slot 0
timed out, connect to ssh
closing fd 4
selecting... max_fd=4 num_probing=0
ssh:connection from 79.117.189.134:58184 to 10.0.0.226:443 forwarded from 127.0.0.1:52890 to 127.0.0.1:22
socket closed
connection closed down

Edit: Playing a bit with sslh, I've noticed that using ssh after starting a connection with openvpn can influence in the detection of openvpn, is that possible?

sslh-1   | ssh:connection from x.x.x.x:58631 to 10.0.0.226:443 forwarded from 127.0.0.1:39198 to 127.0.0.1:22
sslh-1   | ssh:connection from x.x.x.x:58632 to 10.0.0.226:443 forwarded from 127.0.0.1:53520 to 127.0.0.1:22
sslh-1   | ssh:connection from x.x.x.x:58633 to 10.0.0.226:443 forwarded from 127.0.0.1:53526 to 127.0.0.1:22
sslh-1   | ssh:connection from x.x.x.x:58634 to 10.0.0.226:443 forwarded from 127.0.0.1:56024 to 127.0.0.1:22
sslh-1   | hexdump of incoming packet:
sslh-1   | 0x000000: 00 56 38 55 38 d1 77 d8 57 21 83 31 74 12 68 64 .V8U8.w.W!.1t.hd
sslh-1   | 0x000010: 11 d8 b9 36 39 0c 5f 7f 54 6d 4d e7 57 ff 16 21 ...69._.TmM.W..!
sslh-1   | 0x000020: 0c 75 9b 84 ec 29 63 2c 83 02 8a 16 68 a3 5a b4 .u...)c,....h.Z.
sslh-1   | 0x000030: e9 76 ef 9b b7 48 bd cb b7 4a 90 bb cd 4f e3 68 .v...H...J...O.h
sslh-1   | 0x000040: 84 d2 6d 06 b1 da 9b 39 ee 16 df 00 00 00 01 66 ..m....9.......f
sslh-1   | 0x000050: 05 8f fc 00 00 00 00 00 00 56 38 55 38 d1 77 d8 .........V8U8.w.
sslh-1   | 0x000060: 57 21 83 e7 60 12 e3 25 bc 24 d6 bc 82 e6 79 de W!..`..%.$....y.
sslh-1   | 0x000070: 03 47 55 7b be c2 60 d6 73 8e 95 9b f8 58 68 73 .GU{..`.s....Xhs
sslh-1   | 0x000080: 8e 26 b9 fe e6 e6 f4 74 84 6b d2 71 83 f0 b3 48 .&.....t.k.q...H
sslh-1   | 0x000090: 1a 3b f9 e3 9c 2e 93 c9 30 d3 21 3a f0 ed 40 b1 .;......0.!:..@.
sslh-1   | 0x0000a0: 74 a0 83 00 00 00 02 66 05 8f fc 00 00 00 00 00 t......f........
sslh-1   | openvpn:connection from x.x.x.x:58642 to 10.0.0.226:443 forwarded from 127.0.0.1:48284 to 127.0.0.1:4443
sslh-1   | hexdump of incoming packet:
sslh-1   | 0x000000: 53 53 48 2d 32 2e 30 2d 50 75 54 54 59 5f 4b 69 SSH-2.0-PuTTY_Ki
sslh-1   | 0x000010: 54 54 59 0d 0a                                  TTY..
sslh-1   | ssh:connection from x.x.x.x:58655 to 10.0.0.226:443 forwarded from 127.0.0.1:54614 to 127.0.0.1:22

Starting a ssh connection while openvpn it's timing out automatically triggers the detection of openvpn by sslh

Still no packet dump from openvpn, just ssh

yrutschle commented 7 months ago

Yes, it's timing out, but should it at least show the incoming packets

No. What happens is:

sslh accepts an incoming connection.
the remote client sends some data
sslh probes the data to recognise the protocol (and prints the data)
based on the protocol it forwards to the appropriate port

Some rare protocols do not require the client to send data (notably, ssh). In that case, the remote client connects to sslh, and then ... nothing happens (an ssh server would send data). So after some time sslh times out the connection and forwards to the last protocol in the list (or that specified with --on-timeout.).

So, in the case of time out, there is no incoming data to print.

I've noticed that using ssh after starting a connection with openvpn can influence in the detection of openvpn, is that possible?

Not in theory, and unlikely in practice. If you want to be sure, try it out with sslh-fork instead of sslh-select: with forking, each connection is handled by a different process and there is basically no interference between connections at all.

Other than that, what we see is:

sslh probe for openvpn seems to work fine (it recognises it when there is an incoming connection)
some (all? most?) openvpn connections open the TCP socket but sslh receives no data

I would try to investigate with tcpdump on the openvpn side (to check if it's sending anything) and then on the sslh side (to check if it's receiving anything). I remember OpenVPN opens the TCP connection before thinking about how it's going to create cryptography parameters, and creating the random numbers on a system that has little entropy sources can take a long time (I remember, like, minutes). If I remember correctly one way around that is to use haveged. If this theory is true, OpenVPN would also take a long time connecting with sslh out of the loop.

Which actually would be an interesting experience to diagnose the problem: do direct openvpn connection work fine?

scmanjarrez commented 7 months ago

Ok, this is really weird.

sslh-fork always work test1
sslh-select only works after sending a ssh connection, it just stays waiting and then on ssh connection, it connects. test1 test2
no sslh, always work (openvpn listening directly on port 443)

I'm not really sure what's happening here. Also, I've tried looking at the packets, they're are sent and received, but they don't match, the hex shown in sslh isn't matching what I see when I inspect the packets captured with tcpdump. Should TCP frame match? what part of the packet is the hex shown when packet-verbose is enabled?

yrutschle commented 7 months ago

I observe weird behaviours on the very first connections, where the data seems processed after the connection closed. I didn't notice because when testing with t_load, the many connections drown the problem. Would it work fine for you after those few connections?

This happens only with sslh-select, I'll dig to see what's happening.

scmanjarrez commented 7 months ago

It doesn't work unless I do, actively, "modify" the state, using ssh or manually restarting the openvpn connection (this doesn't always work as shown in test). After that first (triggered) connection, every attempt to connect the vpn always work.

test reconnect, openvpn at the end test reconnect, ssh at the end test reconnect, not working the first attempt, ssh at the end

yrutschle commented 7 months ago

There once was
an off-by-one error
t'was fixed badly
and it popped again

Hopefully 1799a8107927704ba9e5d920996297b4cf671256 will fix it right this time. Basically, the connection with the highest file descriptor didn't get served right, which means "occasionally" (but rarely, and the bigger the traffic the server gets, the rare it gets) a connection would not be processed. Interestingly, this bug has been present for 3 years!

scmanjarrez commented 7 months ago

Thank you very much for your hard work! Now it's working flawlessly! 🎉🎉🎉 Edit: It seems that now it (also) always dumps the hex packet when using verbose-packets argument

yrutschle commented 7 months ago

@robonuka feel free to re-open if this does not fix your problem (it's possible, but you didn't specify if you were using sslh-select)

robonuka commented 7 months ago

Thank you @yrutschle for fixing this problem! I tested your native docker image ghcr.io/yrutschle/sslh:latest Now sslh works correct with openvpn and ssh redirections.

yrutschle / sslh

--openvpn does not live together with --ssh option #425