Openvpn GUI crashes. The connection remains.

[MDcod]

Drops randomly. Didn't notice the patterns. The tray icon disappears and the OpenVPN GUI for Windows process disappears. Network connectivity remains good. But now there is no way to break the connection. Just restart the computer. I don't see gui logs anywhere, only the network itself.

OS: Windows 11 Pro 21H2 (22000.1219), also crashed on earlier versions OpenVpn: (v2.5.8), also crashed on earlier versions OpenVpn GUI: (v11.31.0.0), also crashed on earlier versions

Please help, it's incredibly annoying

[MDcod]

Maybe I can give some more useful information about it?

[selvanair]

Maybe I can give some more useful information about it?

Yes please !

[MDcod]

@selvanair What for example? Can the GUI logs be written somewhere? I don't see any errors in the Windows logs when it crashes either.

[selvanair]

OpenVPN logs may help for a start. When the GUI crashes leaving openvpn.exe running, go to OpenVPN\log in your user profile folder and look for a file named <foo>.log where <foo> is the name of the connection that was started. Also look into Windows event logs.

Alternatively, when the GUI crashes, got to task-manager and kill the openvpn.exe process left behind. That will terminate the VPN connection -- no need to reboot. Then restart the GUI and from its menu open the log file of the connection that crashed (connection-name->View Log). Upload it somewhere and post a link here.

Another thing to do is to narrow down the issue to a particular OpenVPN/OS version combination. Your initial post says

OS: Windows 11 Pro 21H2 (22000.1219), also crashed on earlier versions OpenVpn: (v2.5.8), also crashed on earlier versions OpenVpn GUI: (v11.31.0.0), also crashed on earlier versions

That's not helpful. If every version is crashing on all versions of the OS, what can I say.

[mduivcap]

I'm having the same issue as the poster mentioned but with a newer version.

The app does crash when I try to connect en when I try to disconnect. In between it usually does not crash. Unfortunatelly I cannot end the task (company policy).

The logfile:

2023-03-10 15:10:46 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set. 2023-03-10 15:10:46 Note: --cipher is not set. OpenVPN versions before 2.5 defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers. 2023-03-10 15:10:46 Note: '--allow-compression' is not set to 'no', disabling data channel offload. 2023-03-10 15:10:46 OpenVPN 2.6.0 [git:v2.6.0/b999466418dddb89] Windows-MSVC [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] [DCO] built on Jan 25 2023 2023-03-10 15:10:46 Windows version 10.0 (Windows 10 or greater), amd64 executable 2023-03-10 15:10:46 library versions: OpenSSL 3.0.7 1 Nov 2022, LZO 2.10 2023-03-10 15:10:46 MANAGEMENT: TCP Socket listening on [AF_INET]127.0.0.1:25340 2023-03-10 15:10:46 Need hold release from management interface, waiting... 2023-03-10 15:10:46 MANAGEMENT: Client connected from [AF_INET]127.0.0.1:50229 2023-03-10 15:10:47 MANAGEMENT: CMD 'state on' 2023-03-10 15:10:47 MANAGEMENT: CMD 'log on all' 2023-03-10 15:10:47 MANAGEMENT: CMD 'echo on all' 2023-03-10 15:10:47 MANAGEMENT: CMD 'bytecount 5' 2023-03-10 15:10:47 MANAGEMENT: CMD 'state' 2023-03-10 15:10:47 MANAGEMENT: CMD 'hold off' 2023-03-10 15:10:47 MANAGEMENT: CMD 'hold release' 2023-03-10 15:13:17 MANAGEMENT: CMD 'username "Auth" "username"' 2023-03-10 15:13:18 MANAGEMENT: Client disconnected

The weird part is that even when the connection is established, this logfile does not change, it will still say Client disconnected at the end. I know VPN is working when I can reach a certain url.

When you startup openvpngui again, it does not show that there is a connection up and running, so besides the fact that the gui should not crash, it would be nice to at least be able to start it up again and see if there are connections.

Is there anything else we can provide to make this problem more clear?

[selvanair]

When the GUI crashes, can you check the event log for "Application Error" and copy paste the details of that event here. It may give some clue on where the crash is happening.

[mduivcap]

This is what I see when the app crashes when the VPN connects and when I disconnect (not always, but this happens).

Faulting application name: openvpn-gui.exe, version: 11.37.0.0, time stamp: 0x63d2a419 Faulting module name: ntdll.dll, version: 10.0.22000.918, time stamp: 0x57b668f2 Exception code: 0xc0000374 Fault offset: 0x000000000010c509 Faulting process id: 0x34ec Faulting application start time: 0x01d956504777ec85 Faulting application path: C:\Program Files\OpenVPN\bin\openvpn-gui.exe Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll Report Id: 4629660c-7907-4c53-b867-e8fdd7fe36a0 Faulting package full name: Faulting package-relative application ID:

[mduivcap]

Could this be the answer? https://answers.microsoft.com/en-us/windows/forum/all/app-crash-with-ntdlldll/9aa59f80-99a1-4f3c-b2f0-7eb72df05b2a

[selvanair]

Faulted in ntdll.dll does not mean that the error is there. It could still be the application that is causing the dll to crash -- it just makes debugging harder.

If you can provide exact commands you use, any other actions you do that could affect the config files between connect and the crash, I can try to reproduce.

2023-03-10 15:10:47 MANAGEMENT: CMD 'hold release' 2023-03-10 15:13:17 MANAGEMENT: CMD 'username "Auth" "username"' 2023-03-10 15:13:18 MANAGEMENT: Client disconnected

Assuming that "disconnect" is because the GUI crashed, does it always happen after sending username as in the log snippet above?

[mduivcap]

The exact commands I don't know. Connect > Crash What I do in the GUI is, I fill in the password and hit "OK" The crash happens when the connection seems to succeed. Sometimes I see the screen in the picture just disappear (no Windows notification) and when I go to the system tray icon, it's there, but when I hover over it, it disappears (because it crashed). Sometimes it's similar, but I do briefly see the Windows notification popping up (see second picture) And as mentioned, sometimes it does not crash and I see the notification and the system tray Icon remains where it is as it should And just to be clear. When the app crashes it usually did connect the VPN and I usually can visit the url that is behind the VPN even though the system tray icon is gone. So app crash does not equal loosing the connection. But when I reopen the GUI app, the connection does not appear to be there in system tray icon (it will just show you the app like you just started it up with no active connections, but in fact you are connected) Disconnect > Crash What I do is open the system tray, rightclick on the icon, click on the connection that is active, click disconnect. It seems to crash at the moment of disconnecting (when I have the log open, I do see something happening before it crashes and afterwards I'm not connected anymore, so it did disconnect, it just also made the app crash)

[selvanair]

The exact commands I don't know.

I misunderstood that you were sending commands to a running GUI as in a previous crash report on a related thread.

Thanks for the details -- looks like very common and normal usage. The connection remaining after GUI crash is expected as OpenVPN.exe will continue to run. Also, restarting the GUI in such cases will not work right, as it wont be able to connect back to the running OpeVPN.exe process.

But the GUI should not crash in the first place. We'll see whether anyone with access to Windows 11 Pro can reproduce this: ping @lstipakov

[lstipakov]

I have Windows 11 (ARM64, not sure if that matters) but was unable to reproduce this, even though I use gui somewhat actively. Will keep an eye on it.

[mduivcap]

A, good to mention indeed. I'm using Win11 on x64

[WigF1]

The exact commands I don't know.

I misunderstood that you were sending commands to a running GUI as in a previous crash report on a related thread.

Thanks for the details -- looks like very common and normal usage. The connection remaining after GUI crash is expected as OpenVPN.exe will continue to run. Also, restarting the GUI in such cases will not work right, as it wont be able to connect back to the running OpeVPN.exe process.

But the GUI should not crash in the first place. We'll see whether anyone with access to Windows 11 Pro can reproduce this:

I use Win11 Pro per

I have this exact issue, I've had it on Win11 21H2 and Win11 22H2. It can sometimes be 4,5, or 6 attempts before I am able to establish a connection.

I usually end up restarting the OpenVPN Interactive Service, then re-opening the GUI.

Based on the Exception code that appears in the Application event log, it would seem to suggest something other than end-user environments:

https://learn.microsoft.com/en-us/answers/questions/1001963/ntdll-dll-application-error-(0xc0000374)

[chironh]

Experiencing the same issue on Windows 11 Home 22H2 with both 2.6.3 and 2.6.4.

OpenVPN icon just disappears from taskbar, but VPN connection is still active. Only way to kill the connection is to restart the OpenVPN Interactive Service, as mentioned by WigF1.

Faulting application name: openvpn-gui.exe, version: 11.42.0.0, time stamp: 0x645cb914 Faulting module name: ntdll.dll, version: 10.0.22621.1485, time stamp: 0xb7f0154a Exception code: 0xc0000374 Fault offset: 0x000000000010be19 Faulting process id: 0x0xBF4 Faulting application start time: 0x0x1D98C903E1BDAC1 Faulting application path: C:\Program Files\OpenVPN\bin\openvpn-gui.exe Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: 3e595943-d88b-4795-8fae-20cd0d0d7bf2 Faulting package full name: Faulting package-relative application ID:

[Murrh]

Me and my colleague have been getting this same issue on and off for a few months. Win11 Version 22H2 Build 22621.1702

EventID: 1000 MachineName: PC Data : {} Index: 13079 Category: (100) CategoryNumber: 100 EntryType: Error Message: Faulting application name: openvpn-gui.exe, version: 11.41.0.0, time stamp: 0x6437e0e8 Faulting module name: ntdll.dll, version: 10.0.22621.1485, time stamp: 0xb7f0154a Exception code: 0xc0000374 Fault offset: 0x000000000010be19 Faulting process ID: 0x0x2ff0 Faulting application start time: 0x0x1d98c77cf516745 Faulting application path: C:\Program Files\OpenVPN\bin\openvpn-gui.exe Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report ID: 5b05f800-f22a-4f69-bbaf-548882d8744d

Faulting package full name: Faulting package-relative application ID: Source: Application Error ReplacementStrings: {openvpn-gui.exe, 11.41.0.0, 6437e0e8, ntdll.dll...} InstanceId: 1000 TimeGenerated: 5/22/2023 8:37:09 AM TimeWritten: 5/22/2023 8:37:09 AM UserName : PC

The annoying part about using restart-service openvpnserviceinteractive is that it doesn't know about active running VPN's. This results in me having about 10 active VPN's after a while and running out of TAP drivers (and Windows is not very good at multiple tunnels it feels like). If I'm not in a hurry I check the running VPN's with get-netadapter and restart them with restart-netadapter before restarting the GUI so I start with a clean sheet.

[lstipakov]

Do you have any specific steps to reproduce it?

[selvanair]

@lstipakov Can we provide a debug build and collect a crash dump?

In general, setting up a process for crash reporting would be useful. To make it easier to collect crash dumps, shall we enable dumps on an a per application basis during installation? We'll also need pdb files corresponding to release builds.

[chironh]

Do you have any specific steps to reproduce it?

Not really, it seems it just crashes while using.

Most of the time I only notice the crash when I want to disconnect and the icon's missing. Because the connection doesn't drop, it's not very clear "when" or "why" the crash happens.

Is there a setting to enable verbose logs?

[mduivcap]

Do you have any specific steps to reproduce it?

It mainly happens when trying to connect (does not matter if it passes or fails) and when you try to disconnect. I might have seen it crash while it was connected, but I'm not fully sure anymore.

[lstipakov]

I have attached openvpn-gui.exe with debug symbols enabled, openvpn-gui.pdb which contains debug symbols and registry file to enable dump collection for openvpn-gui.exe.

Could you please:

1. Enable dump collection for openvpn-gui.exe by running .\enable-gui-dump.reg under administrative privileges. This should create key HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\openvpn-gui.exe.
2. Copy openvpn-gui.exe to C:\Program Files\OpenVPN\bin.
3. If openvpn-gui is running, close it and rerun.
4. Reproduce the crash.
5. A crash dump should appear at %LOCALAPPDATA%\CrashDumps. Paste it here.

gui-dump.zip

[Alexey-Kovalenko]

Well, I'm not any of above speakers, but have same issue. Openvpn-gui randomly disappear (silently crashes). My keys are password protected. Here is a dump. openvpn-gui.exe.7568.zip

[lstipakov]

Here is a dump

Thanks. Sadly it is not super clear from the dump where is the problem, apart from "heap corruption".

Microsoft (R) Windows Debugger Version 10.0.22621.755 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Temp\dump\gui\openvpn-gui.exe.7568.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available

Symbol search path is: srv*
Executable search path is: 
Windows 10 Version 19045 MP (4 procs) Free x64
Product: WinNt, suite: SingleUserTS
Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
Machine Name:
Debug session time: Sun May 28 18:01:04.000 2023 (UTC + 3:00)
System Uptime: not available
Process Uptime: 0 days 0:01:10.000
.........................................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1d90.abc): Unknown exception - code c0000374 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtWaitForMultipleObjects+0x14:
00007ffe`6366dbb4 c3              ret
0:003> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** WARNING: Unable to verify timestamp for openvpn-gui.exe

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 2937

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 88931

    Key  : Analysis.Init.CPU.mSec
    Value: 3468

    Key  : Analysis.Init.Elapsed.mSec
    Value: 21500

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 129

    Key  : Timeline.Process.Start.DeltaSec
    Value: 70

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Timestamp
    Value: 2019-12-06T14:06:00Z

    Key  : WER.OS.Version
    Value: 10.0.19041.1

    Key  : WER.Process.Version
    Value: 11.42.0.0

FILE_IN_CAB:  openvpn-gui.exe.7568.dmp

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

CONTEXT:  (.ecxr)
rax=00007ffe63744a24 rbx=00000000c0000374 rcx=0000001389efeef0
rdx=0000001389eff3f0 rsi=0000000000000001 rdi=00007ffe637397f0
rip=00007ffe636cf449 rsp=0000001389eff3f0 rbp=0000000000000000
 r8=0000001389eff400  r9=0000000000000000 r10=0000000000000202
r11=0000001389efeef0 r12=0000000000000000 r13=0000013f819e9d50
r14=0000013f819e9d40 r15=0000000000000001
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206
ntdll!RtlReportFatalFailure+0x9:
00007ffe`636cf449 eb00            jmp     ntdll!RtlReportFatalFailure+0xb (00007ffe`636cf44b)
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ffe636cf449 (ntdll!RtlReportFatalFailure+0x0000000000000009)
   ExceptionCode: c0000374
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 00007ffe637397f0

PROCESS_NAME:  openvpn-gui.exe

ERROR_CODE: (NTSTATUS) 0xc0000374 - A heap has been corrupted.

EXCEPTION_CODE_STR:  c0000374

EXCEPTION_PARAMETER1:  00007ffe637397f0

STACK_TEXT:  
00000013`89eff3f0 00007ffe`636cf413     : 00007ffe`6371760c 00000013`89eff8f0 00000000`00000000 0000013f`819e9050 : ntdll!RtlReportFatalFailure+0x9
00000013`89eff440 00007ffe`636d8192     : 00000000`00000001 00007ffe`637397f0 00000000`00000003 0000013f`81970000 : ntdll!RtlReportCriticalFailure+0x97
00000013`89eff530 00007ffe`636d847a     : 00000000`00000003 00000000`00000000 0000013f`81970000 00000000`00000003 : ntdll!RtlpHeapHandleError+0x12
00000013`89eff560 00007ffe`636de101     : 0000013f`81970000 0000013f`81970000 00007ffe`61052c80 00007ffe`60f79af3 : ntdll!RtlpHpHeapHandleError+0x7a
00000013`89eff590 00007ffe`63677482     : 00000000`00000003 00007ffe`62f64449 00000000`00000001 00007ffe`635e9877 : ntdll!RtlpLogHeapFailure+0x45
00000013`89eff5c0 00007ffe`635f47b1     : 00000000`7ffe0385 0000013f`81970000 0000013f`81970000 00000000`00000000 : ntdll!RtlpFreeHeapInternal+0x81d72
00000013`89eff680 00007ffe`6364053c     : 0000013f`819e36d8 00000000`0000000d 00000013`8974c000 00000000`00000000 : ntdll!RtlFreeHeap+0x51
00000013`89eff6c0 00007ffe`635e7665     : 00000013`8974c000 00000000`00000000 00000000`00000000 0000013f`819730c0 : ntdll!LdrpFreeTls+0x84
00000013`89eff6f0 00007ffe`6362467e     : 0000013f`819d2100 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrShutdownThread+0x215
00000013`89eff7f0 00007ffe`6121f3aa     : 00007ffe`60390000 00000000`00000000 0000013f`819d21c0 00000000`00000000 : ntdll!RtlExitUserThread+0x3e
00000013`89eff830 00007ffe`603a20df     : 00000000`00000000 00000000`00000001 00000000`00000001 00007ffe`60390000 : KERNELBASE!FreeLibraryAndExitThread+0x4a
00000013`89eff860 00007ffe`63317614     : ffffffff`ffffffff ffffffff`ffffffff 00000000`00000000 00000000`00000000 : mswsock!SockAsyncThread+0x110cf
00000013`89eff8c0 00007ffe`636226a1     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
00000013`89eff8f0 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

STACK_COMMAND:  ~3s; .ecxr ; kb

SYMBOL_NAME:  mswsock!SockAsyncThread+110cf

MODULE_NAME: mswsock

IMAGE_NAME:  mswsock.dll

FAILURE_BUCKET_ID:  HEAP_CORRUPTION_c0000374_mswsock.dll!SockAsyncThread

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  10.0.19041.546

FAILURE_ID_HASH:  {0e6358ef-db41-8e79-4ff7-0745ef881149}

Followup:     MachineOwner
---------

0:003> kn
 # Child-SP          RetAddr               Call Site
00 00000013`89efe3b8 00007ffe`636add88     ntdll!NtWaitForMultipleObjects+0x14
01 00000013`89efe3c0 00007ffe`636ad36e     ntdll!WerpWaitForCrashReporting+0xa8
02 00000013`89efe440 00007ffe`636acb2b     ntdll!RtlReportExceptionHelper+0x33e
03 00000013`89efe510 00007ffe`636cf484     ntdll!RtlReportException+0x9b
04 00000013`89efe590 00007ffe`6365c876     ntdll!RtlReportFatalFailure$filt$0+0x33
05 00000013`89efe5c0 00007ffe`636723df     ntdll!_C_specific_handler+0x96
06 00000013`89efe630 00007ffe`636214a4     ntdll!RtlpExecuteHandlerForException+0xf
07 00000013`89efe660 00007ffe`636211f5     ntdll!RtlDispatchException+0x244
08 00000013`89efed70 00007ffe`636cf449     ntdll!RtlRaiseException+0x185
09 00000013`89eff3f0 00007ffe`636cf413     ntdll!RtlReportFatalFailure+0x9
0a 00000013`89eff440 00007ffe`636d8192     ntdll!RtlReportCriticalFailure+0x97
0b 00000013`89eff530 00007ffe`636d847a     ntdll!RtlpHeapHandleError+0x12
0c 00000013`89eff560 00007ffe`636de101     ntdll!RtlpHpHeapHandleError+0x7a
0d 00000013`89eff590 00007ffe`63677482     ntdll!RtlpLogHeapFailure+0x45
0e 00000013`89eff5c0 00007ffe`635f47b1     ntdll!RtlpFreeHeapInternal+0x81d72
0f 00000013`89eff680 00007ffe`6364053c     ntdll!RtlFreeHeap+0x51
10 00000013`89eff6c0 00007ffe`635e7665     ntdll!LdrpFreeTls+0x84
11 00000013`89eff6f0 00007ffe`6362467e     ntdll!LdrShutdownThread+0x215
12 00000013`89eff7f0 00007ffe`6121f3aa     ntdll!RtlExitUserThread+0x3e
13 00000013`89eff830 00007ffe`603a20df     KERNELBASE!FreeLibraryAndExitThread+0x4a
14 00000013`89eff860 00007ffe`63317614     mswsock!SockAsyncThread+0x110cf
15 00000013`89eff8c0 00007ffe`636226a1     kernel32!BaseThreadInitThunk+0x14
16 00000013`89eff8f0 00000000`00000000     ntdll!RtlUserThreadStart+0x21

[Alexey-Kovalenko]

At least, I think bug in TCP/IP communications with openvpn.exe, started by service (HEAP_CORRUPTION_c0000374_mswsock.dll!SockAsyncThread) CrashDumps.zip I made two more dumps, running procmon in parallel.

[lstipakov]

The second dump looks more interesting:

00 00000065`323fca28 00007ffe`636add88     ntdll!NtWaitForMultipleObjects+0x14
01 00000065`323fca30 00007ffe`636ad36e     ntdll!WerpWaitForCrashReporting+0xa8
02 00000065`323fcab0 00007ffe`636acb2b     ntdll!RtlReportExceptionHelper+0x33e
03 00000065`323fcb80 00007ffe`636cf484     ntdll!RtlReportException+0x9b
04 00000065`323fcc00 00007ffe`6365c876     ntdll!RtlReportFatalFailure$filt$0+0x33
05 00000065`323fcc30 00007ffe`636723df     ntdll!_C_specific_handler+0x96
06 00000065`323fcca0 00007ffe`636214a4     ntdll!RtlpExecuteHandlerForException+0xf
07 00000065`323fccd0 00007ffe`636211f5     ntdll!RtlDispatchException+0x244
08 00000065`323fd3e0 00007ffe`636cf449     ntdll!RtlRaiseException+0x185
09 00000065`323fda60 00007ffe`636cf413     ntdll!RtlReportFatalFailure+0x9
0a 00000065`323fdab0 00007ffe`636d8192     ntdll!RtlReportCriticalFailure+0x97
0b 00000065`323fdba0 00007ffe`636d847a     ntdll!RtlpHeapHandleError+0x12
0c 00000065`323fdbd0 00007ffe`636de101     ntdll!RtlpHpHeapHandleError+0x7a
0d 00000065`323fdc00 00007ffe`63677482     ntdll!RtlpLogHeapFailure+0x45
0e 00000065`323fdc30 00007ffe`635f47b1     ntdll!RtlpFreeHeapInternal+0x81d72
0f 00000065`323fdcf0 00007ffe`2d322248     ntdll!RtlFreeHeap+0x51
10 00000065`323fdd30 00007ffe`2d2c6a50     riched20!CW32System::FreePv+0x1c
11 00000065`323fdd60 00007ffe`2d313030     riched20!CDisplay::Thaw+0xb4
12 00000065`323fde50 00007ffe`2d2b38c1     riched20!CTxtEdit::SetText+0x6d4
13 00000065`323fe030 00007ffe`2d2b630b     riched20!CTxtEdit::TxSendMessage+0xa41
14 00000065`323fe2e0 00007ffe`6258e858     riched20!RichEditWndProc+0xbeb
15 00000065`323fece0 00007ffe`6258de1b     user32!UserCallWinProcCheckWow+0x2f8
16 00000065`323fee70 00007ffe`6258d68a     user32!SendMessageWorker+0x70b
17 00000065`323fef10 00007ff6`3d7cae07     user32!SendMessageW+0xda
18 00000065`323fef60 00000174`8f49d3ad     openvpn_gui!OnLogLine+0x227 [C:\Users\lev\Projects\openvpn-build\src\openvpn-gui\openvpn.c @ 223] 
19 00000065`323fef68 00000000`00000001     0x00000174`8f49d3ad
1a 00000065`323fef70 00000174`8f449e90     0x1
1b 00000065`323fef78 00007ffe`635f47b1     0x00000174`8f449e90
1c 00000065`323fef80 00000000`00000000     ntdll!RtlFreeHeap+0x51

which points to this line: SendMessage(logWnd, EM_SCROLLCARET, 0, 0);

[selvanair]

Being a heap corruption, its hard to say where the fault really is. It could be anywhere before the crash triggers, and the exception location in the dumps do not look helpful. Is there something like valgrind on Windows that could check invalid reads and writes etc?

[chipitsine]

I like that tool https://github.com/dynamorio/drmemory

[mduivcap]

For what's it worth. I do need to connect to two VPN's and I noticed that with one connection the openvpngui app crashes more often than the other. I'll ask if I'm allowed to share some details about the differences between the two, maybe this gives a hint in where (part of) the problem might be.

[sscotter]

I have around 130 users which connect to our company network using OpenVPN and the OpenVPN GUI. One of those users is regularly having problems with the GUI silently crashing (and as others mention, the VPN stays up.. just the GUI disappears from the system tray). From what I gather it occurs most days, multiple times a day.. but occasionally it'll be fine for a day, then revert to crashing the following day. There's no rhythm nor reason to it. Nothing out of the usual with his machine or his use case compared to any of his colleagues.

The issue was first reported in April 2023 and we've tried the following

• Upgrading to the latest versions are soon as they come out
• Compared and contrasted his machine to his colleagues
• Deleting and recreating his account on the firewall, which results in creating him a new VPN profile (including certificate and key)
• Reinstalled Windows
• Multiple other things I do not recall!

He is part of a team which normally connects to two different VPNs simultaneously. There is around 60 people in that team and no one else has the same issue. He feels the GUI crashes more often than not when connecting to one particular VPN.

When the problem persisted across windows reinstallations I felt the problem must be related to his VPN configuration file but I've since borrowed his VPN config and ran it on my machine over the weekend and didn't have any problems.

Very frustrating and confusing.

If anyone has any suggestions on how to fix this OR how to possibly gain more diagnostic information to try and get to the root case I'd be grateful!

Thanks!

[selvanair]

If anyone has any suggestions on how to fix this OR how to possibly gain more diagnostic information to try and get to the root case I'd be grateful!

Most helpful feedback would be one that could help us reproduce the crash. Would you happen to know from which version of the GUI this started? So far crash dumps have not been particularly useful as it seems to be a heap corruption which is hard to pin down. Especially so when this happens only on some end-user installations.

[lstipakov]

Since 2.6.5 we should have dump collection feature. @sscotter Can you check if there is anything in %LOCALAPPDATA%\CrashDumps ?

[sscotter]

@selvanair .. looking back at my conversations with the user I can see they first reported the issue on 9th February 2023 (not April 2023 as I previously stated). Looking at the software inventory history of the machine it appears it was upgraded from 2.5.8-I604 to 2.6.0-I003 on the 8th at my request for compliance reasons. We can see he went back to 2.5.8 on 13th Feb for two days, presumable in an attempt to restore stability, which I suspect was fruitless because he went back to 2.6.0 on the 15th.

Full timeline of changes of version installed on the machine

Dec 15th 2022
Uninstalled  Software  [OpenVPN 2.5.5-I602 amd64]
Installed Software  [OpenVPN 2.5.8-I604 amd64]

Feb 8th 2023
Uninstalled  Software  [OpenVPN 2.5.8-I604 amd64]
Installed Software  [OpenVPN 2.6.0-I003 amd64]

**** PROBLEM REPORTED HERE **** 

Feb 10th 2023
Uninstalled  Software  [OpenVPN 2.6.0-I003 amd64]
Installed Software  [OpenVPN 2.6.0-I004 amd64]

Feb 13th 2023
Installed Software  [OpenVPN 2.5.8-I604 amd64]
Uninstalled  Software  [OpenVPN 2.6.0-I004 amd64]

Feb 15th 2023
Uninstalled  Software  [OpenVPN 2.5.8-I604 amd64]
Installed Software  [OpenVPN 2.6.0-I003 amd64]

Mar 16th 2023
Uninstalled  Software  [OpenVPN 2.6.0-I003 amd64]
Installed Software  [OpenVPN 2.6.1-I001 amd64]

Mar 30th 2023
Uninstalled  Software  [OpenVPN 2.6.1-I001 amd64]
Installed Software  [OpenVPN 2.6.2-I001 amd64]

Apr 19th 2023
Uninstalled  Software  [OpenVPN 2.6.2-I001 amd64]
Installed Software  [OpenVPN 2.6.3-I001 amd64]

Jul 27th 2023
Uninstalled  Software  [OpenVPN 2.6.3-I001 amd64]
Installed Software  [OpenVPN 2.6.5-I001 amd64]

We've upgraded the machine to 2.6.6 this morning.. unfortunately the problem persists.

---

@lstipakov in regards to your request for the Crash Dumps, we've checked the machine and there is no AppData\Local\CrashDumps directory in the users home directory. There is a CrashDump for another local admin account used to installer software though, but that only contains dumps for msiexec.exe and not related to OpenVPN in any way.

Checking C:\Users\USER\AppData\Local\CrashDumps...
Path does not exist. No crashdumps available.
Checking C:\Users\LOCALADMIN\AppData\Local\CrashDumps...
Path exists! Heres a list of all the crashdumps available.

    Directory: C:\Users\LOCALADMIN\AppData\Local\CrashDumps

Mode                 LastWriteTime         Length Name                                                                 
----                 -------------         ------ ----                                                                 
-a----        26/07/2023     11:45        2993585 msiexec.exe.4328.dmp                                                 
-a----        26/07/2023     11:42        2995437 msiexec.exe.9580.dmp                                                 
Checking C:\Users\Public\AppData\Local\CrashDumps...
Path does not exist. No crashdumps available.

We checked the machines registry, it had keys for openvpn.exe, openvpn-gui.exe and openvpnserv.exe in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps

---

@selvanair and @lstipakov, thank you both for taking the time to respond. It's appreciated!

[lstipakov]

One way would be to git bisect between those releases and pinpoint the exact commit which causes this issue.

PS C:\Users\lev\Projects\openvpn-build\src\openvpn-gui> git bisect bad
Bisecting: 30 revisions left to test after this (roughly 5 steps)
[8b1976c6e35833ec02ee15deba440454cc888283] Increase max size of management password buffer

@sscotter do you think it would be feasible for you to test ~5 gui executables one by one? I can of course provide binaries.

[sscotter]

Sure. The user is desperate to get to the root cause of the issue and will do pretty much anything to get it resolved!

[lstipakov]

Let's start with this one, built from commit 8b1976c. openvpn-gui-8b1976c.zip

[sscotter]

Hi @lstipakov,

Just to give you an update.. we've deployed the 8b1976c build this morning and so far it's behaving. However it's not unheard for it to behave for a day or two, then revert to having regular problems again. We'll let you know quickly if it crashes again, but if you don't hear from us for a day or two it probably means it's behaving.

To put a slight spanner in the works I'm on vacation from tomorrow for a week. If the user is still having problems I'll post their Github handle here so you two can continue to troubleshoot in my absence.

Again, thanks for taking the time to reply and help us troubleshoot this issue.

[sscotter]

The user reported the GUI behaved all day today. It may be because the problem is resolved.. it may be because he was lucky and I'll come back with a vengeance tomorrow. Lets hope it's for the former.

The user has signed up to github today and his handle is j4ypeem. It you receive any messages from him, its to do with this. Thanks!

[lstipakov]

@sscotter Unfortunately I cannot tag @j4ypeem here. If the version https://github.com/OpenVPN/openvpn-gui/commit/8b1976c6e35833ec02ee15deba440454cc888283 still works (or doesn't work), I would like to continue with bisecting and provide the next build.

upd: oh now I can, nice

[j4ypeem]

@lstipakov Thanks for your help with this issue. Since last week I've been running the build you shared with no problems whatsoever.

I noticed there was a comment on that build regarding max size of password. Is there a known password character limit? Would a 30 char password be a likely factor in this issue?

I'm happy to try another version if you want to troubleshoot further.

[lstipakov]

Hi @j4ypeem,

Thanks for the testing, we're moving forward. Here is the next version from commit https://github.com/OpenVPN/openvpn-gui/commit/c354c19e8fbdd193fbdf212af9c74f3b33a65025. Please give it a try. After this one we'll have only 4 steps left to find offensive commit.

openvpn-gui-c354c1.zip

Note that the password mentioned in the commit message is the management interface password, which is normally set by openvpn-gui and not something user controls, except persistent connections / PLAP cases. Anyway, starting from commit https://github.com/OpenVPN/openvpn-gui/commit/8b1976c6e35833ec02ee15deba440454cc888283 there should not be issues with long passwords.

[j4ypeem]

Last night I replaced the GUI with the c354c1 build. This morning I've encountered the GUI crash again. Are there any GUI logs that I can gather that will help troubleshoot?

[lstipakov]

Good, this is expected. We're narrowing down the changelog to find an offensive commit.

Those are remaining suspects:

PS C:\Users\lev\Projects\openvpn-build\src\openvpn-gui> git bisect view --oneline
c354c19 .gitignore: update
53ae284 Update CHANGES.rst
28a5682 Handle CONNECTED,ROUTE_ERROR state message
b4e3e07 Support colouring of WriteStatusLog messages
d47b5f1 Update italian language
b25a24f Update italian language
f7d594d Update italian language
99a0c5d Fix truncated text in Italian
fa82d69 (HEAD) Update Italian language
cea3d25 Update openvpn-gui-res-zh-hans.rc
6398b1d (tag: v11.34.0.0, tag: OpenVPN-2.6_rc1-I001) Bump version to 11.34.0.0
6daee58 Update CHANGES.rst
548aeb0 Improve README
442a48e BUILD.rst: switch to VS2022
e9e9dae Update vcpkg commit id

Please try this one. openvpn-gui-fa82d6.zip

There might be dumps at %LOCALAPPDATA%\CrashDumps.

[j4ypeem]

No dumps in that location.

Will try this version and report back. Probably tomorrow. Thanks

[j4ypeem]

Same happened with this build but no crash dump files again.

[lstipakov]

Thanks, how about openvpn-gui-548aeb.zip ?

[j4ypeem]

Same crash happens with this one. No crash dump files.

[lstipakov]

Hm.. than doesn't make much sense, the rest of commits do not have any code changes. Let's make sure that our baseline is correct. Can you retest the latest one considered as "good"? I have rebuilt it. openvpn-gui-8b1976c-v2.zip

[selvanair]

If 8b1976c is indeed well-behaved, this points to OpenSSL dll updates (vcpkg commit-id change) as the culprit? But, we do not link in OpenSSL in latest versions which also crash.

[lstipakov]

That “vcpkg commit-id” related to GitHub Actions only (the commit message could be more clear). Lähetetty iPhonestaSelva Nair @.***> kirjoitti 23.8.2023 kello 17.39: If 8b1976c is indeed well-behaved, this points to dependency library updates (vcpkg commit-id change) as the culprit?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[selvanair]

That “vcpkg commit-id” related to GitHub Actions only

I see, so all the git-bisect builds you have provided are linked to the same set of dlls..

For memory dumps the user may have to enable it by creating a registry key: HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\openvpn-gui.exe