crashes in ConEmuHk64.dll

[kai-uwe-rommel]

Versions

ConEmu build: 210822 x32/x64 OS version: Windows 10 21H1 19043.1165 x64 Problem then occus with any subprocess (cmd.exe, powershell, ssh ...) regardless of version

Problem description

Since a couple of week, probably since some recent Windows update, I see occassional (but too often to ignore) crashes of subprocesses started from a shell running under ConEmu. In ConEmu, I run FAR (currently 3.0.5858 x64 but also happes with earlier versions), cmd.exe and PowerShell tabs. When I run a subprocess in one of these (happens to all), then sometimes such a subprocess crashes, to me it looks like when it exits. This is not very often but often enough so that it causes problems. For example, in PowerShell I run scripts than build entire Kubernetes clusters and such scripts call dozens of cmd.exe/ssh.exe/scp.exe subprocesses and during the run of such a script, the probability of a crash of such a subprocess approaches 1. This happens only since about 3...4 weeks, before that it never occured. Only regular Windows updates happened, ConEmu and FAR were not changed. Only after I experienced these problems I updated ConEmu and FAR to see if that solves the problem. When I now run the same scripts under the new Windows Terminal instead of ConEmu, no such crashes happen and everything is stable. The Windows eventlog entries for the crashes show ConEmuHk64.dll as the module that causes a 0xC0000005 crash. See below for details of the event log entries.

Steps to reproduce

1. Use ConEmu64 as my Window manager for shells.
2. In one tab run PowerShell or cmd.exe
3. In that shell run a script that runs a sequence of subprocesses
4. one will eventually crash in ConEmuHk64.dll

Actual results

Crashes.

Expected results

No crashes.

Additional files

Each crash creates a sequence of five 1000 and 1001 event log entries:

Name der fehlerhaften Anwendung: ssh.exe, Version: 8.1.0.0, Zeitstempel: 0x5df9d8dd Name des fehlerhaften Moduls: ConEmuHk64.dll, Version: 21.8.22.0, Zeitstempel: 0x6122c975 Ausnahmecode: 0xc0000005 Fehleroffset: 0x00000000000315be ID des fehlerhaften Prozesses: 0x7078 Startzeit der fehlerhaften Anwendung: 0x01d79c220be508c9 Pfad der fehlerhaften Anwendung: C:\Work\ARS\openshift-deployment\tools\ssh.exe Pfad des fehlerhaften Moduls: C:\BIN\FAR3\ConEmu\ConEmuHk64.dll Berichtskennung: 359329ae-ae69-4e1f-a2a9-f622aea579f5 Vollständiger Name des fehlerhaften Pakets: Anwendungs-ID, die relativ zum fehlerhaften Paket ist:

Fehlerbucket , Typ 0 Ereignisname: APPCRASH Antwort: Nicht verfügbar CAB-Datei-ID: 0 Problemsignatur: P1: ssh.exe P2: 8.1.0.0 P3: 5df9d8dd P4: ConEmuHk64.dll P5: 21.8.22.0 P6: 6122c975 P7: c0000005 P8: 00000000000315be P9: P10:

Name der fehlerhaften Anwendung: ssh.exe, Version: 8.1.0.0, Zeitstempel: 0x5df9d8dd Name des fehlerhaften Moduls: ConEmuHk64.dll_unloaded, Version: 21.8.22.0, Zeitstempel: 0x6122c975 Ausnahmecode: 0xc0000005 Fehleroffset: 0x000000000002b200 ID des fehlerhaften Prozesses: 0x7078 Startzeit der fehlerhaften Anwendung: 0x01d79c220be508c9 Pfad der fehlerhaften Anwendung: C:\Work\ARS\openshift-deployment\tools\ssh.exe Pfad des fehlerhaften Moduls: ConEmuHk64.dll Berichtskennung: 20660456-6da2-41ee-b21d-2cfc1338218b Vollständiger Name des fehlerhaften Pakets: Anwendungs-ID, die relativ zum fehlerhaften Paket ist:

Fehlerbucket 1231080142501493285, Typ 4 Ereignisname: APPCRASH Antwort: Nicht verfügbar CAB-Datei-ID: 0 Problemsignatur: P1: ssh.exe P2: 8.1.0.0 P3: 5df9d8dd P4: ConEmuHk64.dll P5: 21.8.22.0 P6: 6122c975 P7: c0000005 P8: 00000000000315be P9: P10:

Fehlerbucket 2180596766113371912, Typ 5 Ereignisname: BEX64 Antwort: Nicht verfügbar CAB-Datei-ID: 0 Problemsignatur: P1: ssh.exe P2: 8.1.0.0 P3: 5df9d8dd P4: ConEmuHk64.dll_unloaded P5: 21.8.22.0 P6: 6122c975 P7: 000000000002b200 P8: c0000005 P9: 0000000000000008 P10: Angefügte Dateien: \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER6D4B.tmp.dmp \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER6D9A.tmp.WERInternalMetadata.xml \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER6DAA.tmp.xml \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER6DBA.tmp.csv \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER6DDA.tmp.txt Diese Dateien befinden sich möglicherweise hier: \?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_ssh.exe_f18ce9eaee48795346140bd18a18c7c3d627df2_818c891b_54cebdc5-edeb-449e-abeb-89ebd97ed00f Analysesymbol: Es wird erneut nach einer Lösung gesucht: 0 Berichts-ID: 20660456-6da2-41ee-b21d-2cfc1338218b Berichtstatus: 268435456 Bucket mit Hash: a8591ba7bacbedf24e4309349a8d9708 CAB-Datei-Guid: 0

WER-files.zip

[Maximus5]

Could you provide any crash dump? This would simplify investigations

[kai-uwe-rommel]

I had attached WER files. Other dump files were not created.

[kai-uwe-rommel]

Anything more I can do? Do you have any idea yet?

[Maximus5]

It's hard to say what happens on your PC. Just a few observations:

Two provided .wer files have different offsets. Looks like a random location of crashes.

Your ssh.exe has loaded the C:\Program Files (x86)\Citrix\ICA Client\epclient64.dll. That most probably means that Citrix injects the epclient64.dll library into the ssh.exe process and it somehow interferes with ConEmuHk64.dll. If you can (have right, etc.) it would be interesting if you could add ssh.exe to some exclusions for Citrix (I'm not sure what it's doing).

Another thing you could try (to create a crash dumps) is if the crash happens quite often, you may call in your prompt set ConEmuReportExe=ssh.exe and try to reproduce. When ConEmu shows the message "ssh.exe loaded" you may execute tab context menu "Debug active process". After the debugger tab is started, you may close the "ssh.exe loaded" dialog. If a crash happens, the debugger then will create a crash dump.

For now, I may only guess that "Citrix" has some updates last time which is the reason for your problems. Probably, that could be solved only by Citrix, but definitely, I cannot be sure without dumps.

As a workaround, I can suggest running your scripts with the -cur_console:i switch which will exclude loading of ConEmuHk64.dll in your processes.

[kai-uwe-rommel]

Hmm, Citrix ... first I will simply uninstall it (I can get by without it for a while) and see if the problem disappears so that I know that it really is caused by Citrix. Then I'll test with the latest Citrix client version (my current one is from last year).

[kai-uwe-rommel]

With Citrix uninstalled, I had no crashes in a sequence of scripts where I otherwise would have had a couple of these.

[Maximus5]

Then we definitely have conflicted detours somehow. Could you try to create a dumps I've asked?

[kai-uwe-rommel]

I'll reinstall a current Citrix Client version and if it then crashes again, I will try to create dump. Is there a way to configure that a dump should be created whenever a process crashes? Because I do not know it advance when/if a subprocess will crash. It's perhaps been one out of 20 or so.

[Maximus5]

You may try postmortem debugger from sysinternals. https://conemu.github.io/en/MemoryDump.html#Using_Postmortem_Debugger

[kai-uwe-rommel]

So far, with the fresh installed new Citrix version, no crashes yet. However, I have omitted its "App Protection" feature which might exactly be the component that introduces this Citrix hook DLL.