OpenVPN stopped working, no error messages

ProfaneServitor commented 2 months ago

I live in Crimea. I used the same configuration of OpenVPN for last 4 years or so.

I used this script to install OpenVPN:

https://github.com/angristan/openvpn-install

2 days ago it mysteriously stopped working. My client can connect to VPN server, but when I ping 8.8.8.8 while connected to it, all packets are lost.

Here is tail -f /var/log/syslog output when I connect and try pinging anything:

Aug 22 23:23:17 VPS ovpn-server[676]: MULTI: multi_create_instance called
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Re-using SSL/TLS context
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Control Channel MTU parms [ L:1621 D:1156 EF:94 EB:0 ET:0 EL:3 ]
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Data Channel MTU parms [ L:1621 D:1450 EF:121 EB:406 ET:0 EL:3 ]
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1549,tun-mtu 1500,proto UDPv4,cipher AES-128-GCM,auth [null-digest],keysize 128,key-method 2,tls-server'
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1549,tun-mtu 1500,proto UDPv4,cipher AES-128-GCM,auth [null-digest],keysize 128,key-method 2,tls-client'
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> TLS: Initial packet from [AF_INET]<SERVER_IP>, sid=f0fe61af ab705f83
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> VERIFY OK: depth=1, CN=cn_ZEa1Gfis95ePD3Y4
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> VERIFY OK: depth=0, CN=<CLIENT_ID>
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_VER=2.4.7
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_PLAT=linux
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_PROTO=2
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_NCP=2
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_LZ4=1
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_LZ4v2=1
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_LZO=1
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_COMP_STUB=1
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_COMP_STUBv2=1
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> peer info: IV_TCPNL=1
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 256 bit EC, curve prime256v1, signature: ecdsa-with-SHA256
Aug 22 23:23:17 VPS ovpn-server[676]: <SERVER_IP> [<CLIENT_ID>] Peer Connection Initiated with [AF_INET]<SERVER_IP>
Aug 22 23:23:17 VPS ovpn-server[676]: MULTI: new connection by client '<CLIENT_ID>' will cause previous active sessions by this client to be dropped.  Remember to use the --duplicate-cn option if you want multiple clients using the same certificate or username to concurrently connect.
Aug 22 23:23:17 VPS ovpn-server[676]: MULTI_sva: pool returned IPv4=10.8.0.2, IPv6=(Not enabled)
Aug 22 23:23:17 VPS ovpn-server[676]: MULTI: Learn: 10.8.0.2 -> <CLIENT_ID>/<SERVER_IP>
Aug 22 23:23:17 VPS ovpn-server[676]: MULTI: primary virtual IP for <CLIENT_ID>/<SERVER_IP>: 10.8.0.2
Aug 22 23:23:17 VPS ovpn-server[676]: Outgoing Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Aug 22 23:23:17 VPS ovpn-server[676]: Incoming Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Aug 22 23:23:19 VPS ovpn-server[676]: <CLIENT_ID>/<SERVER_IP> PUSH: Received control message: 'PUSH_REQUEST'
Aug 22 23:23:19 VPS ovpn-server[676]: <CLIENT_ID>/<SERVER_IP> SENT CONTROL [<CLIENT_ID>]: 'PUSH_REPLY,dhcp-option DNS 8.8.8.8,redirect-gateway def1 bypass-dhcp,route-gateway 10.8.0.1,topology subnet,ping 10,ping-restart 120,ifconfig 10.8.0.2 255.255.255.0,peer-id 1,cipher AES-128-GCM' (status=1)
Aug 22 23:23:19 VPS ovpn-server[676]: <CLIENT_ID>/<SERVER_IP> PID_ERR replay-window backtrack occurred [1] [SSL-0] [0_00] 0:4 0:3 t=1724368999[0] r=[0,64,15,1,1] sl=[60,4,64,528]
Aug 22 23:23:19 VPS ovpn-server[676]: <CLIENT_ID>/<SERVER_IP> MULTI: bad source address from client [<CLIENT_IP>], packet dropped
Aug 22 23:23:19 VPS ovpn-server[676]: message repeated 3 times: [ <CLIENT_ID>/<SERVER_IP> MULTI: bad source address from client [<CLIENT_IP>], packet dropped]

/etc/openvpn/server.conf:

port 57373
proto udp
dev tun
user nobody
group nogroup
persist-key
persist-tun
keepalive 10 120
topology subnet
server server_ip 255.255.255.0
float
duplicate-cn
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 94.140.14.14"
push "dhcp-option DNS 94.140.15.15"
push "redirect-gateway def1 bypass-dhcp"
dh none
ecdh-curve prime256v1
tls-crypt tls-crypt.key
crl-verify crl.pem
ca ca.crt
cert server_stuff.crt
key server_stuff.key
auth SHA256
cipher AES-128-GCM
ncp-ciphers AES-128-GCM
tls-server
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
client-config-dir /etc/openvpn/ccd
status /var/log/openvpn/status.log
verb 3

journalctl -u NetworkManager.service:

Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.0994] device (tun1): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1040] device (tun1): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1053] device (tun1): Activation: starting connection 'tun1' (a476129f-bdf0-481c-ae82-ddb803ace8ba)
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1054] device (tun1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1058] device (tun1): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1062] device (tun1): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1065] device (tun1): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1127] policy: set 'kservers' (tun1) as default for IPv4 routing and DNS
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1593] device (tun1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1596] device (tun1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
Aug 29 20:51:36 cherry NetworkManager[874]: <info>  [1724953896.1616] device (tun1): Activation: successful, device activated.

I called my ISP, and they claim they didn't block anything. Also, shareware VPN providers like SetupVPN and hoxxcom still work, so I doubt it's the government doing it. What else can I do to diagnose or fix this problem?

EDIT:

I live in Crimea, so Russian government might interfere. The server is in Netherlands.

ooVizieRoo commented 2 months ago

Hello. Did you solve the issue?

I have the same troubles. Only change to TCP proto in conf make it work but unstable.

ordex commented 1 month ago

@ProfaneServitor if I get the log right it seems the server is getting a new connection and it is reporting his own IP as source IP? You have edited the client IP in several places with "SERVER IP". This is a bit confusing.

ProfaneServitor commented 3 weeks ago

@ooVizieRoo No, I didn't solve anything. I have been using freemium browsers VPNs for months.

@ordex I edited the ips out because I don't want to be doxed. Can you elaborate what's the problem with server reporting the outcoming ip as its own? Shouldn't it do it by default?

cron2 commented 4 days ago

Your log file is a server log (the "multi" lines wouldn't be in a client log) - so the OpenVPN server would normally never log client-related lines with SERVER_IP. If your terminology is "this is the IP of the server in our office which is connecting to this OpenVPN server instance", this might make sense, but it's hard for us to follow. In an OpenVPN server log, there are no "server" IPs in a tls handshake.

That said, the handshake is all fine, and there are no errors in the log. It's possible that NetworkManager ate part of your config (it's ignoring the redirect-gateway def1 from the server, because NM always knows better). It's also possible that routing/NAT/firewalling config on the OpenVPN server was lost/messed-up.

What you need to do now is

run ping 8.8.8.8 on the client
do tcpdump/wireshark on the client tun interface (tcpdump -n -s0 -i tun1 according to the log) to see if packets go in the tunnel - if not, it's a local routing issue on the client
do tcpdump/wireshark on the server tun interface, to see if packets come out of the tunnel - if not, it's an ISP/filtering issue on the path. If they come out, and no responses come back, it's a routing/firewalling/NAT issue on the server side, beyond OpenVPN's reach

I will close this issue now, as there is no indication that there is an OpenVPN bug involved. User help requests should go to the openvpn-users@ mailing list, or to the forum. This place is for bug reports.

OpenVPN / openvpn

OpenVPN stopped working, no error messages #597