search

OpenVPN / openvpn

OpenVPN is an open source VPN daemon
http://openvpn.net
Other
11.07k stars 3.02k forks source link

crash on OpenVPN 2.6.2 Ubuntu 22.04 LTS #301

Closed knitdv closed 1 year ago

knitdv commented 1 year ago

Please tell me, I can't install the ovpn-dco-v2 kernel module.

/home/paul/ovpn-dco/gen-compat-autoconf.sh /home/paul/ovpn-dco/compat-autoconf.h
make -C /lib/modules/5.15.0-67-generic/build M=/home/paul/ovpn-dco PWD=/home/paul/ovpn-dco REVISION=0.2.20230323-1-g1c2c84e CONFIG_OVPN_DCO_V2=m INSTALL_MOD_DIR=updates/   modules
make[1]: Entering directory '/usr/src/linux-headers-5.15.0-67-generic'
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/main.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/bind.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/crypto.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/ovpn.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/peer.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/sock.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/stats.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/netlink.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/crypto_aead.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/pktid.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/tcp.o
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/udp.o
  LD [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/ovpn-dco-v2.o
  MODPOST /home/paul/ovpn-dco/Module.symvers
  CC [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/ovpn-dco-v2.mod.o
  LD [M]  /home/paul/ovpn-dco/drivers/net/ovpn-dco/ovpn-dco-v2.ko
  BTF [M] /home/paul/ovpn-dco/drivers/net/ovpn-dco/ovpn-dco-v2.ko
Skipping BTF generation for /home/paul/ovpn-dco/drivers/net/ovpn-dco/ovpn-dco-v2.ko due to unavailability of vmlinux
make[1]: Leaving directory '/usr/src/linux-headers-5.15.0-67-generic'
root@vpn-cl:/home/paul/ovpn-dco# make install
/home/paul/ovpn-dco/gen-compat-autoconf.sh /home/paul/ovpn-dco/compat-autoconf.h
make -C /lib/modules/5.15.0-67-generic/build M=/home/paul/ovpn-dco PWD=/home/paul/ovpn-dco REVISION=0.2.20230323-1-g1c2c84e CONFIG_OVPN_DCO_V2=m INSTALL_MOD_DIR=updates/ modules_install
make[1]: Entering directory '/usr/src/linux-headers-5.15.0-67-generic'
arch/x86/Makefile:142: CONFIG_X86_X32 enabled but no binutils support
  INSTALL /lib/modules/5.15.0-67-generic/updates//drivers/net/ovpn-dco/ovpn-dco-v2.ko
  SIGN    /lib/modules/5.15.0-67-generic/updates//drivers/net/ovpn-dco/ovpn-dco-v2.ko
At main.c:167:
- SSL error:FFFFFFFF80000002:system library::No such file or directory: ../crypto/bio/bss_file.c:67
- SSL error:10000080:BIO routines::no such file: ../crypto/bio/bss_file.c:75
sign-file: certs/signing_key.pem: No such file or directory
  DEPMOD  /lib/modules/5.15.0-67-generic
Warning: modules_install: missing 'System.map' file. Skipping depmod.
make[1]: Leaving directory '/usr/src/linux-headers-5.15.0-67-generic'
depmod -a

I tried to run the module manually modprobe ovpn-dco-v2

root@vpn-cl:/home/paul/ovpn-dco# modprobe ovpn-dco-v2
modprobe: ERROR: could not insert 'ovpn_dco_v2': Operation not permitted

My OS and kernel version: 

root@vpn-cl:/home/paul/ovpn-dco# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:    22.04
Codename:   jammy
root@vpn-cl:/etc/openvpn# uname -r
5.15.0-67-generic

Tell me how you can solve the problem , or the best practice of which OS to install so that there are no problems with OpenVPN 2.6.2 with DCO ?

Thanks.

schwabe commented 1 year ago

What does dmesg report when you try the modprobe?

knitdv commented 1 year ago

When entering the dmesg command , I received Lockdown: modprobe: unsigned module loading is restricted; see man kernel_lockdown.7

I disabled UEFI secure boot in bios, the module loaded

OpenVPN server and client have started, everything works with the disable_dco option, without errors.

With the dco module, there are errors in the client's work.

Server config:

port 1100
proto tcp-server
dev tun_vpn
local 1.1.1.1
mode server
topology subnet
tls-server

ca      /etc/openvpn/easy-rsa/pki/ca.crt
cert    /etc/openvpn/easy-rsa/pki/issued/server.crt
key     /etc/openvpn/easy-rsa/pki/private/server.key  # keep secret
dh      /etc/openvpn/easy-rsa/pki/dh.pem
tls-auth /etc/openvpn/server/ta.key 0

client-config-dir ccd/

server 10.10.1.0 255.255.255.248  # internal tun0 connection IP

keepalive 10 20
auth-nocache

#auth SHA1 
cipher ChaCha20-Poly1305
data-ciphers ChaCha20-Poly1305

persist-key
persist-tun

status vpn_gw.log
log-append  vpn_gw.log

verb 3  # verbose mode

#explicit-exit-notify 1

Client config:


client
dev tun
proto tcp-client
#disable-dco
#local  1.1.1.2
tls-client

remote 1.1.1.1 1100         # [VPN server IP] [PORT]
#resolv-retry infinite
nobind

persist-key
persist-tun

ca      vpn_cl/ca.crt
cert    vpn_cl/vpn_cl.crt
key     vpn_cl/vpn_cl.key
tls-auth vpn_cl/ta.key 1

remote-cert-tls server
auth-nocache

#cipher AES-256-CBC
#data-ciphers AES-256-CBC

cipher ChaCha20-Poly1305
data-ciphers ChaCha20-Poly1305

mute-replay-warnings

verb 3
status vpn_cl.log
log-append  vpn_cl.log

Logs:

Server

2023-03-27 01:37:01 OpenVPN 2.6.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2023-03-27 01:37:01 library versions: OpenSSL 3.0.2 15 Mar 2022, LZO 2.10
2023-03-27 01:37:01 DCO version: 0.2.20230323-1-g1c2c84e
2023-03-27 01:37:01 net_route_v4_best_gw query: dst 0.0.0.0
2023-03-27 01:37:01 net_route_v4_best_gw result: via 0.0.0.0 dev
2023-03-27 01:37:01 Diffie-Hellman initialized with 2048 bit key
2023-03-27 01:37:01 net_route_v4_best_gw query: dst 0.0.0.0
2023-03-27 01:37:01 net_route_v4_best_gw result: via 0.0.0.0 dev
2023-03-27 01:37:01 ROUTE_GATEWAY 0.0.0.0
2023-03-27 01:37:01 net_iface_new: add tun_vpn type ovpn-dco
2023-03-27 01:37:01 DCO device tun_vpn opened
2023-03-27 01:37:01 net_iface_mtu_set: mtu 1500 for tun_vpn
2023-03-27 01:37:01 net_iface_up: set tun_vpn up
2023-03-27 01:37:01 net_addr_v4_add: 10.10.1.1/29 dev tun_vpn
2023-03-27 01:37:01 Could not determine IPv4/IPv6 protocol. Using AF_INET
2023-03-27 01:37:01 Socket Buffers: R=[131072->131072] S=[16384->16384]
2023-03-27 01:37:01 Listening for incoming TCP connection on [AF_INET]1.1.1.1:1100
2023-03-27 01:37:01 TCPv4_SERVER link local (bound): [AF_INET]1.1.1.1:1100
2023-03-27 01:37:01 TCPv4_SERVER link remote: [AF_UNSPEC]
2023-03-27 01:37:01 MULTI: multi_init called, r=256 v=256
2023-03-27 01:37:01 IFCONFIG POOL IPv4: base=172.40.1.2 size=5
2023-03-27 01:37:01 MULTI: TCP INIT maxclients=1024 maxevents=1029
2023-03-27 01:37:01 Initialization Sequence Completed
2023-03-27 01:37:13 TCP connection established with [AF_INET]1.1.1.2:40062
2023-03-27 01:37:13 1.1.1.2:40062 Connection reset, restarting [0]
2023-03-27 01:37:13 1.1.1.2:40062 SIGUSR1[soft,connection-reset] received, client-instance restarting
2023-03-27 01:37:18 TCP connection established with [AF_INET]1.1.1.2:55510
2023-03-27 01:37:18 1.1.1.2:55510 Connection reset, restarting [0]
2023-03-27 01:37:18 1.1.1.2:55510 SIGUSR1[soft,connection-reset] received, client-instance restarting
2023-03-27 01:37:23 TCP connection established with [AF_INET]1.1.1.2:55514
2023-03-27 01:37:23 1.1.1.2:55514 Connection reset, restarting [0]

Client

2023-03-27 01:43:15 OpenVPN 2.6.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2023-03-27 01:43:15 library versions: OpenSSL 3.0.2 15 Mar 2022, LZO 2.10
2023-03-27 01:43:15 DCO version: 0.2.20230323-1-g1c2c84e
2023-03-27 01:43:15 TCP/UDP: Preserving recently used remote address: [AF_INET]1.1.1.1:1100
2023-03-27 01:43:15 Socket Buffers: R=[131072->131072] S=[16384->16384]
2023-03-27 01:43:15 Attempting to establish TCP connection with [AF_INET]1.1.1.1:1100
2023-03-27 01:43:15 TCP connection established with [AF_INET]1.1.1.1:1100
2023-03-27 01:43:15 TCPv4_CLIENT link local: (not bound)
2023-03-27 01:43:15 TCPv4_CLIENT link remote: [AF_INET]1.1.1.1:1100

Client command dmesg

[   19.427010] ovpn_dco_v2: module verification failed: signature and/or required key missing - tainting kernel
[   19.427375] OpenVPN data channel offload (ovpn-dco) 0.2.20230323-1-g1c2c84e -- (C) 2020-2023 OpenVPN, Inc.
[   26.701156] loop5: detected capacity change from 0 to 8
[   71.582292] show_signal_msg: 22 callbacks suppressed
[   71.582295] openvpn[1431]: segfault at 70 ip 0000555c8d6a97f1 sp 00007ffe78fb9488 error 4 in openvpn[555c8d69c000+99000]
[   71.582302] Code: 89 f5 53 48 89 fb 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 44 24 08 31 c0 e8 7a 4f ff ff 48 85 c0 0f 84 a4 00 00 00 6a 00 <8b> 4b 18 45 31 c9 45 31 c0 55 31 d2 31 f6 48 89 c7 49 89 c4 e8 b6
[   76.849471] openvpn[1435]: segfault at 70 ip 0000560b7d3a07f1 sp 00007ffea5bc2538 error 4 in openvpn[560b7d393000+99000]
[   76.849482] Code: 89 f5 53 48 89 fb 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 44 24 08 31 c0 e8 7a 4f ff ff 48 85 c0 0f 84 a4 00 00 00 6a 00 <8b> 4b 18 45 31 c9 45 31 c0 55 31 d2 31 f6 48 89 c7 49 89 c4 e8 b6
[   82.084650] openvpn[1439]: segfault at 70 ip 000055d70ece37f1 sp 00007ffd5417fac8 error 4 in openvpn[55d70ecd6000+99000]

Can you tell me what's wrong now?

schwabe commented 1 year ago

We need a proper backtrace to diagnose the problem and to understand where the OpenVPN binary is crashing. Running OpenVPN under gdb to get a backtrace is one option.

selvanair commented 1 year ago

If this 2.6.2 binary is from OpenVPN apt repo, running under gdb may not provide an informative backtrace as the binary is stripped. I would hazard a guess that this is the same segfault as I reported in https://patchwork.openvpn.net/project/openvpn2/patch/20230327171236.51771-1-selva.nair@gmail.com/

selvanair commented 1 year ago

I cross-checked the segfault info in the dmesg posted here with what I get for the get_doc_peer_stats() segfault using the 2.6.2 binary from the deb package for jammy. The crash location (instruction pointer) matches: 0xd7f1 from the VMA starting address. Can't get much more info than that as the binary has no relevant symbols.

I think the crash in this case is triggered by print_status() which also calls dco_get_peer_stats(). No sure why it gets called before open_tun() though.

Edit: confirmed that print_status() accessing dco_get_peer_stats() can happen very early and causes segfault. No need for management or restart to trigger this.

selvanair commented 1 year ago

@kolya25 Could you please test by removing status vpn_cl.log from client config? That should bypass a bug fixed since 2.6.2.

knitdv commented 1 year ago

I deleted the status vpn_cl.log from client config, errors remained in the server and client logs as before

selvanair commented 1 year ago

I deleted the status vpn_cl.log from client config, errors remained in the server and client logs as before

I never saw any errors in your server or client logs: the server logged TCP disconnect which is not an "error". Client logs end abruptly with no errors logged. The segfault in dmesg was the only indication of an error. I guess you are not running this from command line -- otherwise you would have got Segmentation Fault on command line. So impossible to know whatever is executing openvpn is adding ---status or any other options.

Either wait for 2.6.3 or build OpenVPN from the latest master branch and try. Then post client logs at verb 4 and stack trace using a non-stripped binary if you still get a segfault.

knitdv commented 1 year ago

After the 2.6.3 update , the error disappeared .Thanks.

ogolovanov commented 10 months ago

Latest ubuntu server 22.04 with latest kernel available ( 5.15.0-92.102 )

Using ovpn_dco_v2 kernel module does not work for me ( DCO ignored ).

ogolovanov@server:~$ lsmod | grep dco ovpn_dco_v2 73728 0 ip6_udp_tunnel 16384 1 ovpn_dco_v2 udp_tunnel 20480 1 ovpn_dco_v2

Syslog:

Jan 31 03:54:40 server kernel: [ 569.744536] OpenVPN data channel offload (ovpn-dco) ovpn:0-20220601git2db65af -- (C) 2020-2022 OpenVPN, Inc. Jan 31 03:54:40 server kernel: [ 569.744547] ovpn: can't register RTNL link ops Jan 31 03:54:40 server kernel: [ 569.744680] ovpn: initialization failed, error status=-17

Connection established, but DCO ignored.

Jan 31 03:54:40 server openvpn3-service-logger[2651]: {tag:1046415159024048155} Connected: xxx:1194 (xxx) via /UDPv4 on tun/10.8.0.5/ gw=[10.8.0.1/]

Google knows nothing about error shown above. At the same time old kernel module "ovpn_dco" works as expected ( package = "kmod-ovpn-dco" ).

lsmod | grep dco ovpn_dco 73728 0 ip6_udp_tunnel 16384 1 ovpn_dco udp_tunnel 20480 1 ovpn_dco

Syslog:

Jan 31 04:14:11 server openvpn3-service-logger[2651]: {tag:8058288393314618067} Connected: : (xxx) via /UDPv4-DCO on ovpn-dco/10.8.0.5/ gw=[10.8.0.1/]

I really hope that someday it will work without "dancing".

ordex commented 10 months ago

Are you running openvpn-2.6.3 or later? From what you are saying it seems you are trying to use a new DCO kernel module against an older OpenVPN userspace software.

ogolovanov commented 10 months ago

Hi.

From what you are saying it seems you are trying to use a new DCO kernel module against an older OpenVPN userspace software.

When i am starting session with DCO enabled -> kernel module "ovpn_dco_v2" is autoloaded. But it does not work ( = connection works, but DCO disabled ). If i manually unload ovpn_dco_v2 and load ovpn_dco -> everything works. So i am confused, why "ovpn_dco_v2" not working for me with latest software on ubuntu 22.04.

Are you running openvpn-2.6.3 or later?

Yes, latest openvpn available.

client:

OpenVPN3/Linux v20 (openvpn3)
OpenVPN core v3.7.2 linux x86_64 64-bit
Copyright (C) 2012-2022 OpenVPN Inc. All rights reserved.

server:

OpenVPN 2.6.8 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
library versions: OpenSSL 3.0.2 15 Mar 2022, LZO 2.10
DCO version: 0.2.20231117
Originally developed by James Yonan
Copyright (C) 2002-2023 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto_ofb_cfb=yes enable_dco=auto enable_dco_arg=auto enable_debug=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=no enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_maintainer_mode=no enable_management=yes enable_option_checking=no enable_pam_dlopen=no enable_pedantic=no enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_werror=no enable_win32_dll=yes enable_wolfssl_options_h=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_openssl_engine=auto with_sysroot=no
dsommers commented 10 months ago

@ogolovanov You need OpenVPN 3 Linux v21 to have ovpn-dco-v2 support.

ordex commented 10 months ago

You are using the openvpn3-linux client. It started supporting ovpn-dco-v2 starting from v21, but you have v20 installed.

Are you also having problems on the server? Sorry but from your message it is not clear what error belongs to what.

ogolovanov commented 10 months ago

Thanks :)

Are you also having problems on the server?

No, there are no problems on the server. Server uses ovpn_dco_v2.

Sorry but from your message it is not clear what error belongs to what.

There were almost no error messages except following error on client:

Jan 31 03:54:40 server kernel: [ 569.744547] ovpn: can't register RTNL link ops
Jan 31 03:54:40 server kernel: [ 569.744680] ovpn: initialization failed, error status=-17