search

OpenVPN / openvpn

OpenVPN is an open source VPN daemon
http://openvpn.net
Other
10.93k stars 3.01k forks source link

Azure Gateway and OpenVPN versions past 2.5.8 incompatible (being investigated by Azure Gateway production team) #236

Closed mbell85 closed 11 months ago

mbell85 commented 1 year ago

I've noticed that when my Windows 11 OpenVPN clients upgraded this week to the Jan. 25 release that the legacy service (auto-start it at boot to enable VPN to Azure Gateway) will no longer start. They all use the OpenVPN 64-bit MSI.

selvanair commented 1 year ago

Legacy service (OpenVPNServiceLegacy) has been deprectaed long time ago. Use its replacement called OpenVPNService backed by openvpnserv2.exe. Its installed by default. Try sc start OpenVPNService. Also the auto-start config must be in the config-auto directory (C:\Program Files\OpenVPN\config-auto by default).

mbell85 commented 1 year ago

@selvanair I appreciate this. I didn't realize this was the case! Cheers! I assume this works in the same way the legacy service does (auto-starts prior to login)?

mbell85 commented 1 year ago

After updating, I moved the configuration file where you said, but it doesn't look like the OpenVPNService is installed. Any ideas?

--

Mark Bell IT Administrator Journeys in Community Living 1130 Haley Rd. Murfreesboro, TN 37129 615-890-4389, ext. 45 (ofc) 615-295-3046 (cell) www.journeystn.orghttp://www.journeystn.org/ www.fb.com/journeysincommunityhttp://www.fb.com/journeysincommunity www.twitter.com/journeystnhttp://www.twitter.com/journeystn

NOTICE: This email may contain confidential (including but not limited to) HIPAA-protected and/or privileged information intended only for specific, predetermined recipients. If you are not the intended recipient, you are hereby notified that any review, further dissemination, distribution or duplication of this communication is STRICTLY FORBIDDEN. Please delete and/or destroy all copies of this message after notifying Mark Bell of the error by reply email or calling 615-295-3046.

From: Selva Nair @.> Sent: Thursday, February 2, 2023 9:31 AM To: OpenVPN/openvpn @.> Cc: Mark Bell @.>; Author @.> Subject: Re: [OpenVPN/openvpn] Legacy service broken on all Windows 11 clients after Jan. 25 release (Issue #236)

[EXTERNAL SENDER] Handle with care! DO NOT open attachments or click links from unknown senders or unexpected email! This email originated from outside the journeystn.org domain!

Legacy service (OpenVPNServiceLegacy) has been deprectaed long time ago. Use its replacement called OpenVPNService backed by openvpnserv2.exe. Its installed by default. Try sc start OpenVPNService. Also the auto-start config must be in the config-auto directory (C:\Program Files\OpenVPN\config-auto by default).

- Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOpenVPN%2Fopenvpn%2Fissues%2F236%23issuecomment-1413929386&data=05%7C01%7Cmark.bell%40journeystn.org%7C9698dc6d1c6340163f6708db0532702e%7C4cd5b4aec470481f8624901f42efe3ed%7C0%7C0%7C638109486426527846%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BrU1%2F49zMz7X37nDR6CQ9itoeTYNBWoR83%2F12XTd2R4%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKN3WZCCVFPLTHDUQ34ORE3WVPHJ5ANCNFSM6AAAAAAUPFGKHE&data=05%7C01%7Cmark.bell%40journeystn.org%7C9698dc6d1c6340163f6708db0532702e%7C4cd5b4aec470481f8624901f42efe3ed%7C0%7C0%7C638109486426527846%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XGkdWP4OsIxqcEIMgDMfpMT8viwayaQGJdIuUs9Ogso%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.**@.>>

NOTICE: This email may contain confidential (including but not limited to) HIPAA-protected and/or privileged information intended only for specific, predetermined recipients. If you are not the intended recipient, you are hereby notified that any review, further dissemination, distribution or duplication of this communication IS STRICTLY FORBIDDEN. Please delete and/or destroy all copies of this message after notifying JICL IT of the error by reply @.***> or by calling 615-890-4389.

mbell85 commented 1 year ago

Looks like removing OpenVPN from the system, rebooting and adding back corrected this part of my issue.

--

Mark Bell IT Administrator Journeys in Community Living 1130 Haley Rd. Murfreesboro, TN 37129 615-890-4389, ext. 45 (ofc) 615-295-3046 (cell) www.journeystn.orghttp://www.journeystn.org/ www.fb.com/journeysincommunityhttp://www.fb.com/journeysincommunity www.twitter.com/journeystnhttp://www.twitter.com/journeystn

NOTICE: This email may contain confidential (including but not limited to) HIPAA-protected and/or privileged information intended only for specific, predetermined recipients. If you are not the intended recipient, you are hereby notified that any review, further dissemination, distribution or duplication of this communication is STRICTLY FORBIDDEN. Please delete and/or destroy all copies of this message after notifying Mark Bell of the error by reply email or calling 615-295-3046.

From: Mark Bell @.> Sent: Thursday, February 2, 2023 9:46 AM To: OpenVPN/openvpn @.> Cc: Mark Bell @.>; Your activity @.> Subject: Re: [OpenVPN/openvpn] Legacy service broken on all Windows 11 clients after Jan. 25 release (Issue #236)

[EXTERNAL SENDER] Handle with care! DO NOT open attachments or click links from unknown senders or unexpected email! This email originated from outside the journeystn.org domain!

--

Mark Bell IT Administrator Journeys in Community Living 1130 Haley Rd. Murfreesboro, TN 37129 615-890-4389, ext. 45 (ofc) 615-295-3046 (cell) www.journeystn.orghttp://www.journeystn.org/<http://www.journeystn.org%3chttp:/www.journeystn.org/> www.fb.com/journeysincommunityhttp://www.fb.com/journeysincommunity<http://www.fb.com/journeysincommunity%3chttp:/www.fb.com/journeysincommunity> www.twitter.com/journeystnhttp://www.twitter.com/journeystn<http://www.twitter.com/journeystn%3chttp:/www.twitter.com/journeystn>

NOTICE: This email may contain confidential (including but not limited to) HIPAA-protected and/or privileged information intended only for specific, predetermined recipients. If you are not the intended recipient, you are hereby notified that any review, further dissemination, distribution or duplication of this communication is STRICTLY FORBIDDEN. Please delete and/or destroy all copies of this message after notifying Mark Bell of the error by reply email or calling 615-295-3046.

From: Selva Nair @.<mailto:@.>> Sent: Thursday, February 2, 2023 9:31 AM To: OpenVPN/openvpn @.<mailto:@.>> Cc: Mark Bell @.<mailto:@.>>; Author @.<mailto:@.>> Subject: Re: [OpenVPN/openvpn] Legacy service broken on all Windows 11 clients after Jan. 25 release (Issue #236)

[EXTERNAL SENDER] Handle with care! DO NOT open attachments or click links from unknown senders or unexpected email! This email originated from outside the journeystn.org domain!

Legacy service (OpenVPNServiceLegacy) has been deprectaed long time ago. Use its replacement called OpenVPNService backed by openvpnserv2.exe. Its installed by default. Try sc start OpenVPNService. Also the auto-start config must be in the config-auto directory (C:\Program Files\OpenVPN\config-auto by default).

- Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOpenVPN%2Fopenvpn%2Fissues%2F236%23issuecomment-1413929386&data=05%7C01%7Cmark.bell%40journeystn.org%7C9698dc6d1c6340163f6708db0532702e%7C4cd5b4aec470481f8624901f42efe3ed%7C0%7C0%7C638109486426527846%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BrU1%2F49zMz7X37nDR6CQ9itoeTYNBWoR83%2F12XTd2R4%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKN3WZCCVFPLTHDUQ34ORE3WVPHJ5ANCNFSM6AAAAAAUPFGKHE&data=05%7C01%7Cmark.bell%40journeystn.org%7C9698dc6d1c6340163f6708db0532702e%7C4cd5b4aec470481f8624901f42efe3ed%7C0%7C0%7C638109486426527846%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XGkdWP4OsIxqcEIMgDMfpMT8viwayaQGJdIuUs9Ogso%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.**@.mailto:***@***.******@***.***>>

NOTICE: This email may contain confidential (including but not limited to) HIPAA-protected and/or privileged information intended only for specific, predetermined recipients. If you are not the intended recipient, you are hereby notified that any review, further dissemination, distribution or duplication of this communication IS STRICTLY FORBIDDEN. Please delete and/or destroy all copies of this message after notifying JICL IT of the error by reply @.<mailto:@.>> or by calling 615-890-4389.

- Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOpenVPN%2Fopenvpn%2Fissues%2F236%23issuecomment-1413953364&data=05%7C01%7Cmark.bell%40journeystn.org%7C74f430b9a9ca4fafe27f08db05349b38%7C4cd5b4aec470481f8624901f42efe3ed%7C0%7C0%7C638109495761902887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lpQHp3ZkbUusHFAr9iazWa2zeDHIoGVmJwjgCnBaHsU%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKN3WZDVMUOBDZDODJ2CJLLWVPJEDANCNFSM6AAAAAAUPFGKHE&data=05%7C01%7Cmark.bell%40journeystn.org%7C74f430b9a9ca4fafe27f08db05349b38%7C4cd5b4aec470481f8624901f42efe3ed%7C0%7C0%7C638109495761902887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F564oYHDwRDsM2jovx4kc0%2FalgGGYV%2BuMrc0KZkVNy4%3D&reserved=0. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>

NOTICE: This email may contain confidential (including but not limited to) HIPAA-protected and/or privileged information intended only for specific, predetermined recipients. If you are not the intended recipient, you are hereby notified that any review, further dissemination, distribution or duplication of this communication IS STRICTLY FORBIDDEN. Please delete and/or destroy all copies of this message after notifying JICL IT of the error by reply @.***> or by calling 615-890-4389.

selvanair commented 1 year ago

The config files are automatically moved by the installer if it finds auto-start configs in use -- i.e. OpenVPNService was running before upgrade. That service has been available since 2016 when the old one was renamed as legacy and deprecated.

but it doesn't look like the OpenVPNService is installed

What does sc query OpenVPNService show? Does C:\Program Files\OpenVPN\bin\openvpnserv2.exe exist? If the service is running you have to restart it for it to recognize changes to the config-auto folder.

mbell85 commented 1 year ago

Thanks for this. I believe Webroot may be interacting with my install, which is causing the issue. Going to exempt the installation folder and see if that helps. I did get OpenVPNService showing up in list of services now, but it looks like the Wintun install was perhaps blocked by Webroot. Again, going to exempt and see if that helps.

mbell85 commented 1 year ago

OK. SO now with that out of the way it is installed and running, but I'm noticing the connection is dropping over and over. Same config that worked with legacy service is not working with the OpenVPN service for some reason. Any ideas on what I should try next would be appreciated. I will check out the Azure guide for connecting via OpenVPN to the gateway in the meantime. Perhaps something has changed?

mbell85 commented 1 year ago

Not sure why it isn't working, still. I do notice in the log that it looks like TCP_CLIENT link local is saying (not bound). Is this Wintun still not working right?

2023-02-02 10:55:27 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication 2023-02-02 10:55:27 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication 2023-02-02 10:55:27 MANAGEMENT: >STATE:1675356927,RESOLVE,,,,,, 2023-02-02 10:55:28 TCP/UDP: Preserving recently used remote address: [AF_INET]XX.XXX.XXX.XXX:443 2023-02-02 10:55:28 ovpn-dco device [OpenVPN Data Channel Offload] opened 2023-02-02 10:55:28 TCP_CLIENT link local: (not bound) 2023-02-02 10:55:28 TCP_CLIENT link remote: [AF_INET]XX.XXX.XXX.XXX:443 2023-02-02 10:55:28 MANAGEMENT: >STATE:1675356928,WAIT,,,,,, 2023-02-02 10:55:28 MANAGEMENT: >STATE:1675356928,AUTH,,,,,, 2023-02-02 10:55:28 TLS: Initial packet from [AF_INET]XX.XXX.XXX.XXX:443, sid=XXXXXXXXXXXX 2023-02-02 10:55:28 Connection reset, restarting [-1] 2023-02-02 10:55:28 Closing DCO interface 2023-02-02 10:55:28 SIGUSR1[soft,connection-reset] received, process restarting 2023-02-02 10:55:28 MANAGEMENT: >STATE:1675356928,RECONNECTING,connection-reset,,,,, 2023-02-02 10:55:28 Restart pause, 300 second(s) 2023-02-02 10:58:13 SIGTERM[hard,init_instance] received, process exiting

lstipakov commented 1 year ago

No that's DCO, the new driver. Somehow it is not able to establish TCP connection with remote. Does it work if you add disable-dco to the config?

lstipakov commented 1 year ago

If that works with disable-dco, would be nice to get logs from the driver. Would you be able to provide logs following this instruction? TCP (and UDP) is surely supposed to work.

mbell85 commented 1 year ago

Will check now and report back.

mbell85 commented 1 year ago

No dice. Still doesn't work with disable-dco added. Same exact config works with legacy service on versions up until 2.6.0.

mbell85 commented 1 year ago

Legacy service (OpenVPNServiceLegacy) has been deprectaed long time ago. Use its replacement called OpenVPNService backed by openvpnserv2.exe. Its installed by default. Try sc start OpenVPNService. Also the auto-start config must be in the config-auto directory (C:\Program Files\OpenVPN\config-auto by default).

Can confirm OpenVPNService is now there and config is in the config-auto directory, but no longer connecting. When trying to use the GUI to connect, I am running into issues from prior posts.

mbell85 commented 1 year ago

Here is log with disable-dco enabled.

2023-02-02 12:12:44 OpenVPN 2.6.0 [git:v2.6.0/b999466418dddb89] Windows-MSVC [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] [DCO] built on Jan 25 2023 2023-02-02 12:12:44 Windows version 10.0 (Windows 10 or greater), amd64 executable 2023-02-02 12:12:44 library versions: OpenSSL 3.0.7 1 Nov 2022, LZO 2.10 2023-02-02 12:12:44 MANAGEMENT: TCP Socket listening on [AF_INET]127.0.0.1:25340 2023-02-02 12:12:44 Need hold release from management interface, waiting... 2023-02-02 12:12:44 MANAGEMENT: Client connected from [AF_INET]127.0.0.1:50261 2023-02-02 12:12:44 MANAGEMENT: CMD 'state on' 2023-02-02 12:12:44 MANAGEMENT: CMD 'log on all' 2023-02-02 12:12:44 MANAGEMENT: CMD 'echo on all' 2023-02-02 12:12:44 MANAGEMENT: CMD 'bytecount 5' 2023-02-02 12:12:44 MANAGEMENT: CMD 'state' 2023-02-02 12:12:44 MANAGEMENT: CMD 'hold off' 2023-02-02 12:12:44 MANAGEMENT: CMD 'hold release' 2023-02-02 12:12:44 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication 2023-02-02 12:12:44 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication 2023-02-02 12:12:44 MANAGEMENT: >STATE:1675361564,RESOLVE,,,,,, 2023-02-02 12:12:45 TCP/UDP: Preserving recently used remote address: [AF_INET]XX.XXX.XXX.XXX:443 2023-02-02 12:12:45 Socket Buffers: R=[65536->65536] S=[65536->65536] 2023-02-02 12:12:45 Attempting to establish TCP connection with [AF_INETXX.XXX.XXX.XXX:443 2023-02-02 12:12:45 MANAGEMENT: >STATE:1675361565,TCP_CONNECT,,,,,, 2023-02-02 12:12:45 TCP connection established with [AF_INET]XX.XXX.XXX.XXX:443 2023-02-02 12:12:45 TCPv4_CLIENT link local: (not bound) 2023-02-02 12:12:45 TCPv4_CLIENT link remote: [AF_INET]XX.XXX.XXX.XXX:443 2023-02-02 12:12:45 MANAGEMENT: >STATE:1675361565,WAIT,,,,,, 2023-02-02 12:12:45 MANAGEMENT: >STATE:1675361565,AUTH,,,,,, 2023-02-02 12:12:45 TLS: Initial packet from [AF_INET]XX.XXX.XXX.XXX:443, sid=b3e2cede 4de09285 2023-02-02 12:12:45 Connection reset, restarting [0] 2023-02-02 12:12:45 SIGUSR1[soft,connection-reset] received, process restarting 2023-02-02 12:12:45 MANAGEMENT: >STATE:1675361565,RECONNECTING,connection-reset,,,,, 2023-02-02 12:12:45 Restart pause, 1 second(s)

mbell85 commented 1 year ago

I just rolled back to 2.5.8 and used the OpenVPNService instead of the legacy one and it connects without issue, so it is something with the newest version that my Azure VPN Gateway is not liking.

selvanair commented 1 year ago

The only thing the service does is to start openvpn executable and watch it so that it can restart if it exits. So this error is unrelated to the service.

Likely some incompatibility between 2.6 client and your server in azure. Could you post the server log at verb 4 to see whether that provides any clue?

mbell85 commented 1 year ago

The only thing the service does is to start openvpn executable and watch it so that it can restart if it exits. So this error is unrelated to the service.

Likely some incompatibility between 2.6 client and your server in azure. Could you post the server log at verb 4 to see whether that provides any clue?

Correct. I moved back to the last version and used the OpenVPNService in lieu of sticking with the legacy service (now knowing that is deprecated) and it works fine on 2.5.8. I will pull those logs and get them up here as soon as I have a chance.

I do appreciate your time.

lstipakov commented 1 year ago

At least we know that DCO seem to not be the cause :)

Microsoft has their own OpenVPN server implementation, so it would be really interesting to know if there is some compatibility issue.

mbell85 commented 1 year ago

Hey guys. Sorry. I've been busy provisioning some new devices and working on some other things. I haven't forgotten about this, though. Still going to dig up those logs for you.

mbell85 commented 1 year ago

So far, this is the only log I'm able to pull on the Azure Gateway:

[ERR] [default] [OVPN_9AFE60A3-B776-C1CC-1D0B-5114AB0FF4F0] Connection failed with error 0x3E3 => The I/O operation has been aborted because of either a thread exit or an application request.

I may need to open a ticket with Microsoft to get anything deeper than that. Would you like me to proceed?

schwabe commented 1 year ago

Yes. There is a known problem with Azure and getting into contact with Microsoft is probably the only way to get it resolved. Someone with an azure gateway account could do a git bisect to figure out which commit exactly broke the service.

mbell85 commented 1 year ago

Schwabe: So for now, we should just stick with 2.5.8?

OpenVPN has made the administration of adding endpoints to our Azure network a breeze up until this point.

From a security standpoint, are we missing out on much versus 2.6.0?

mbell85 commented 1 year ago

Just created the ticket with Azure Support, so hopefully, we'll get some movement on it soon.

mbell85 commented 1 year ago

FYI. Azure Engineer replied to my request that they are aware of the issue and production team is working on it. They've added me to a list of customers affected. Thanks for the guidance on this issue. 🙏

UnoSD commented 1 year ago

you may want to add also the Linux label as it happens to me on Arch with NetworkManager-openvpn (see the duplicate just above this)

schwabe commented 1 year ago

The platform labels are more if the issue is platform specific, which this issue is not.

mbell85 commented 1 year ago

Anyone heard anything new on this? Azure Support decided to close my ticket after adding me to list of impacted tenants, though I can technically open at any time. We've got about 32 endpoints sitting on 2.5.8 still.

jmpohl commented 1 year ago

Looking for answers on this too if anyone has additional info

mbell85 commented 1 year ago

I've heard nothing new lately. Was added to a list of customers, or at least was told I was by Azure support and then asked if they could close the ticket, noting it could take a while to see some movement.

lstipakov commented 1 year ago

@mbell85 Could you please try again? We were told by MSFT that 2.6 compatibility issue is now resolved.

mbell85 commented 1 year ago

Yes. I will try updating to the latest version on an endpoint to test this and report back to you.

mbell85 commented 1 year ago

Nope. I take it back. The latest versions are not working. When I go to connect on version 2.6.5 I get an error saying that the management option can not be parsed. Will attach log.

mbell85 commented 1 year ago

Correction. 2.5.8 does indeed appear to work. But 2.6.5 (latest version) does not. I'll test around a bit more to see which version it stops working on again.

lstipakov commented 1 year ago

This might be a different issue, so would be interesting to look at the log.

mbell85 commented 1 year ago

I was running out of time towards the end of my day but I do plan to do some more testing to see which version I can get to before I experience anymore breaking changes. I'll definitely post up the logs.

mbell85 commented 1 year ago

Still not working starting with 2.6.0.

Here is the log:

2023-07-27 20:12:57 OpenVPN 2.6.0 [git:v2.6.0/b999466418dddb89] Windows-MSVC [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] [DCO] built on Feb 15 2023 2023-07-27 20:12:57 Windows version 10.0 (Windows 10 or greater), amd64 executable 2023-07-27 20:12:57 library versions: OpenSSL 3.0.8 7 Feb 2023, LZO 2.10 2023-07-27 20:12:57 MANAGEMENT: TCP Socket listening on [AF_INET]127.0.0.1:25340 2023-07-27 20:12:57 Need hold release from management interface, waiting... 2023-07-27 20:12:58 MANAGEMENT: Client connected from [AF_INET]127.0.0.1:57477 2023-07-27 20:12:58 MANAGEMENT: CMD 'state on' 2023-07-27 20:12:58 MANAGEMENT: CMD 'log on all' 2023-07-27 20:12:58 MANAGEMENT: CMD 'echo on all' 2023-07-27 20:12:58 MANAGEMENT: CMD 'bytecount 5' 2023-07-27 20:12:58 MANAGEMENT: CMD 'state' 2023-07-27 20:12:58 MANAGEMENT: CMD 'hold off' 2023-07-27 20:12:58 MANAGEMENT: CMD 'hold release' 2023-07-27 20:12:58 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication 2023-07-27 20:12:58 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication 2023-07-27 20:12:58 MANAGEMENT: >STATE:1690506778,RESOLVE,,,,,, 2023-07-27 20:12:58 TCP/UDP: Preserving recently used remote address: [AF_INET]XX.XXX.XXX.XXX:443 2023-07-27 20:12:58 ovpn-dco device [OpenVPN Data Channel Offload] opened 2023-07-27 20:12:58 TCP_CLIENT link local: (not bound) 2023-07-27 20:12:58 TCP_CLIENT link remote: [AF_INET]XX.XXX.XXX.XXX:443 2023-07-27 20:12:58 MANAGEMENT: >STATE:1690506778,WAIT,,,,,, 2023-07-27 20:12:58 MANAGEMENT: >STATE:1690506778,AUTH,,,,,, 2023-07-27 20:12:58 TLS: Initial packet from [AF_INET]XX.XXX.XXX.XXX:443, sid=XXXXXXXXXXXXXX 2023-07-27 20:12:58 VERIFY OK: depth=2, C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA 2023-07-27 20:12:58 VERIFY OK: depth=1, C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA 2023-07-27 20:12:58 VERIFY KU OK 2023-07-27 20:12:58 Validating certificate extended key usage 2023-07-27 20:12:58 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication 2023-07-27 20:12:58 VERIFY EKU OK 2023-07-27 20:12:58 VERIFY X509NAME OK: C=US, ST=Washington, L=Redmond, X.vpn.azure.com 2023-07-27 20:12:58 VERIFY OK: depth=0, C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=X.vpn.azure.com 2023-07-27 20:12:58 Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256 X.vpn.azure.com] Peer Connection Initiated with [AF_INET]XX.XXX.XXX.XXX:443 2023-07-27 20:12:58 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1 2023-07-27 20:12:58 TLS: tls_multi_process: initial untrusted session promoted to trusted 2023-07-27 20:12:59 MANAGEMENT: >STATE:1690506779,GET_CONFIG,,,,,, 2023-07-27 20:12:59 SENT CONTROL [X.vpn.azure.com]: 'PUSH_REQUEST' (status=1) 2023-07-27 20:13:00 PUSH: Received control message: 'PUSH_REPLY,route 192.168.X.X 255.255.X.X,route 10.X.X.X 255.X.X.X,route 10.X.X.X 255.X.X.X,route 10.X.X.X 255.X.X.X,route-gateway 10.66.XX.XX,topology subnet,ifconfig 10.XX.XX.XX 255.XX.XX.XXX,dhcp-option DNS 10.XX.XX.XX.4,cipher AES-256-GCM' 2023-07-27 20:13:00 OPTIONS IMPORT: --ifconfig/up options modified 2023-07-27 20:13:00 OPTIONS IMPORT: route options modified 2023-07-27 20:13:00 OPTIONS IMPORT: route-related options modified 2023-07-27 20:13:00 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified 2023-07-27 20:13:00 OPTIONS IMPORT: data channel crypto options modified 2023-07-27 20:13:00 OPTIONS IMPORT: Server did not request DATA_V2 packet format required for data channel offload 2023-07-27 20:13:00 OPTIONS ERROR: pushed options are incompatible with data channel offload. Use --disable-dco to connect to this server 2023-07-27 20:13:00 ERROR: Failed to apply push options 2023-07-27 20:13:00 Failed to open tun/tap interface 2023-07-27 20:13:00 Closing DCO interface 2023-07-27 20:13:00 SIGUSR1[soft,process-push-msg-failed] received, process restarting 2023-07-27 20:13:00 MANAGEMENT: >STATE:1690506780,RECONNECTING,process-push-msg-failed,,,,, 2023-07-27 20:13:00 Restart pause, 1 second(s) 2023-07-27 20:13:01 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication 2023-07-27 20:13:01 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication 2023-07-27 20:13:01 MANAGEMENT: >STATE:1690506781,RESOLVE,,,,,, 2023-07-27 20:13:01 TCP/UDP: Preserving recently used remote address:

mbell85 commented 1 year ago

Looks like the same error from before. Did MS specify whether disable-dco is needed (or anything else for that matter)? I plan on opening another ticket with them to ask myself.

lstipakov commented 1 year ago

This is a different error - we are past TLS handshake now. Here server hasn't pushed peer-id, which is required for DATA_V2 format. This doesn't look right, since this has been a default behavior for many years - I think starting from 2.4 client pushes IV_PROTO with IV_PROTO_DATA_V2 bit set and server replies with peer-id in PUSH_REPLY and uses DATA_V2 format.

Let me ask MSFT again.

I would also recommend to use 2.6.5 since there were quite a few fixes to DCO implementation, although those are probably irrelevant to this specific issue.

cron2 commented 1 year ago

Indeed, progress!

Now we are at the point where a 2.6.5 client started with --disable-dco should work, while before that, we didn't even reach the "we could do something with options" stage.

Lack of support for DATA_V2 was no problem before DCO because both variants where "equally fine". With DCO, the new kernel data paths do no longer support the old framing. Reasoning behind this: DOC also requires AEAD ciphers. AEAD ciphers capability was introduced with 2.4.0, and P_DATA_V2 for clients was introduced somewhere in the 2.3 timeframe (commit 0e1fd332, November 2014). So all official OpenVPN sources that support AEAD also support DATA_V2, and thus, all combinations that would support DCO also need to support DATA_V2.

Which hints at MSFT using a totally different code basis, and not following upstream developments very closely...

mbell85 commented 1 year ago

Guys. Thank you so much! Would you recommend then that I use 2.6.5 with --disable-dco or stick with 2.5.8 for now? I appreciate your expertise and assistance with this! I think most of us responsible for systems at any organization want to always stay current with updates, including third party ones, especially when they are serving a function as important as OpenVPN does.

mbell85 commented 1 year ago

Just confirming that 2.6.5, with --disable-dco added to the config, does indeed work. Which is the more secure option at this point? To stick with 2.5.8 without --disable-dco or to move to 2.6.5 with --disable-dco enabled?

cron2 commented 1 year ago

Both 2.5.9 and 2.6.5 are fully maintained releases, and should be equally stable and secure. Without --disable-dco, 2.6 is much faster, and uses less CPU at the same time - but if DCO cannot be used, it does not really matter today.

At some point, OpenSSL 1.1.1 will become unsupported, and the OpenSSL 3.0 support in OpenVPN 2.5 is not as good as in 2.6 - at that time, moving to 2.6 might be a good idea, regardless of DCO.

mbell85 commented 1 year ago

Thank you for that explanation! Makes perfect sense.

mbell85 commented 11 months ago

Hey guys. I'm not sure if this is still related but some of my endpoints on 2.6.6 and above are sometimes getting APIPA addresses with OpenVPN. This was not occurring with 2.5.5. Strange thing is it only appears to be happening to a handful of endpoints.

cron2 commented 11 months ago

Logfile or it didn't happen... seriously, OpenVPN Clients will normally apply what the server gives them (PUSH_REPLY), so unless there's 169.254.* in the server reply, it's hard to see how it could end up on the interface.

mbell85 commented 11 months ago

Logfile or it didn't happen... seriously, OpenVPN Clients will normally apply what the server gives them (PUSH_REPLY), so unless there's 169.254.* in the server reply, it's hard to see how it could end up on the interface.

Can't get the log file until I physically go to the location, but it's happened on several of our remote endpoints the past couple of weeks... Here's a screenshot of one with the APIPA address showing up where normally the 10.66.X.X network is showing up on other remote endpoints.

Screenshot 2023-11-13 112408

mbell85 commented 11 months ago

I have gone ahead and run a reset command on the Azure VPN Gateway this morning and all other endpoints appear to be working OK. Needless to say, I've never had this behavior on any of our machines before the beginning of last week. The only change was that endpoints were updated to 2.6.7.

TinCanTech commented 11 months ago

Can't get the log file until I physically go to the location

~You can verify what the server pushes to the client in the server log, it may help.~

Not applicable to Azure.

mbell85 commented 11 months ago

The only thing I'm seeing in the Azure Gateway logs that may be related are a bunch of these, which appear to have ceased since I reset the Gateway:

[ERR] [default] [OVPN_XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX] Connection failed with error 0x3E3 => The I/O operation has been aborted because of either a thread exit or an application request.

mbell85 commented 11 months ago

The vast majority of the logs end like this: Connection successful. Username=***N-ClientsChild IP=10.66.XX.XX, which are obviously successful connections. What could be happening on some of them I wonder where they end up with an APIPA address instead, even if the server is giving out a legit/correct IP?