Symbols of cygwin1.dll are wrong

[johannesthoma]

Hi, its me again :)

Just what I observe when working with a simple dump of a hello world file (I am using the binary version you published): I looks as if the symbols of the cygwin1.dll are read incorrectly. Here is the output:

(gdb) bt                                                                        
#0  0x00007ffcfbf002b4 in ntdll!ZwWaitForMultipleObjects () from C:\Windows\System32\ntdll.dll                                                                  
#1  0x00007ffcf80dcc7e in WaitForMultipleObjectsEx () from C:\Windows\System32\KERNELBASE.dll                                                                   
#2  0x00007ffcf80dcb6e in WaitForMultipleObjects () from C:\Windows\System32\KERNELBASE.dll                                                                     
#3  0x00007ffced165fe0 in cygwin1!.assert () from C:\cygwin64\bin\cygwin1.dll   
#4  0x00007ffced1d6feb in sched_getscheduler () from C:\cygwin64\bin\cygwin1.dll
#5  0x00007ffced1d73c9 in sched_getscheduler () from C:\cygwin64\bin\cygwin1.dll
#6  0x00007ffced2aa1cb in timegm () from C:\cygwin64\bin\cygwin1.dll            
#7  0x00000007ffffcd30 in ?? ()                                                 
#8  0x0000000a00000410 in ?? ()                                                 
#9  0x00000007ffffcdf0 in ?? ()                                                 
#10 0x0000000000000000 in ?? ()                                                 

The ntdll.dll seems to be interpreted correctly but the cygwin1.dll symbols seem to be wrong. I already tried installing cygwin1.dll with debug info (from the cygwin homepage) and also removed all other cygwin1.dll on the file system.

If you need more information, please let me know.

Thanks a lot,

• Johannes

[johannesthoma]

Hi just tried to open a crash dump (as produced by a BSOD) gdb could not open it:

"C:\cygwin64\home\Administrator\dumps/minidump.dmp" is not a core dump: file format not recognized

I also tried to convert the file to Minidump format but without luck so far. I will keep you updated.

[ssbssa]

Could you maybe attach the various files (minidump, executable+debuginfo+source, cygwin1.dll+debuginfo) here?

[johannesthoma]

Sure:

cygwin1-issue7-files.tar.gz

Please let me know if you need further info/files.

Thank you for looking into this.

[ssbssa]

It also doesn't show me the correct backtrace for the dump files, but I get the same result when I attach to the process, so it's not about the minidump.

Then I tried it with cygwin-gdb, without debug info of cygwin1.dll it also doesn't show me a full backtrace when attaching, but with debug info it does:

(gdb) bt
#0  0x00007ffd5226db34 in ntdll!ZwWaitForMultipleObjects () from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffd4fce1d20 in WaitForMultipleObjectsEx () from /cygdrive/c/Windows/System32/KERNELBASE.dll
#2  0x00007ffd4fce1c1e in WaitForMultipleObjects () from /cygdrive/c/Windows/System32/KERNELBASE.dll
#3  0x00007ffcd9975fe0 in cygwait (object=object@entry=0x0, timeout=timeout@entry=0x7ffffcb68, mask=mask@entry=11) at /usr/src/debug/cygwin-3.5.1-1/winsup/cygwin/cygwait.cc:75
#4  0x00007ffcd99e6feb in clock_nanosleep (clk_id=clk_id@entry=1, flags=flags@entry=0, rqtp=rqtp@entry=0x7ffffcbf0, rmtp=rmtp@entry=0x7ffffcc00) at /usr/src/debug/cygwin-3.5.1-1/winsup/cygwin/signal.cc:130
#5  0x00007ffcd99e73c9 in sleep (seconds=<optimized out>) at /usr/src/debug/cygwin-3.5.1-1/winsup/cygwin/signal.cc:174
#6  0x00007ffcd9aba1cb in _sigfe () at sigfe.s:36
#7  0x00000001004010ad in main (argc=1, argv=0xa00000450) at hello.c:8

So I guess the cygwin gdb must have special knowledge to be able to do this. It seems there are some extra patches applied, and the first of those sounds suspicious.

[johannesthoma]

You are right ... the gdb that comes with cygwin even prints the source line of the cygwin build. I will try to combine the relevant cygwin patches with your patches and check if the resulting gdb can analyze my hello.exe with the hello3.dmp.

[johannesthoma]

Hi the last 2 cygwin patches (0005 and 0006) of the cygwin patches don't apply, I am trying without them (I agree that 0001 is the interesting patch). Building just now...

[ssbssa]

With the first patch applied it works for me for hello3.dmp and hello3-fulldump.dmp, but not for the others. What's the difference between them?

[johannesthoma]

Hi, sorry I should remove the old ones. I took those with earlier versions of the hello.exe program. Please ignore them.

[ssbssa]

Ah, ok, that might explain this.

[johannesthoma]

Cool good to see that the cygwin patch is working with your patch ... so

gdb hello.exe hello3.dmp

also displays the correct cygwin1.dll backtrace, right? If yes then thanks a lot and I think we can close this issue.

I am still compiling with the patch ... but if you can share your binary or even make a release that would help a lot (if you would include the cygwin patch 0001 in your branch .. don't know if this is the right thing to do because it probably eventually breaks the cygwin build process..).

[ssbssa]

Cool good to see that the cygwin patch is working with your patch ... so

gdb hello.exe hello3.dmp

also displays the correct cygwin1.dll backtrace, right? If yes then thanks a lot and I think we can close this issue.

Well, almost. To get a proper backtrace I also needed the correct versions of ntdll.dll and kernelbase.dll from the microsoft symbol server.

If you have the necessary dlls (dbghelp/dbgcore/symsrv) and put them besides gdb, then my build can get them automatically:

(gdb) set minidump-search-path .;srv*C:\src\dlls-syms*https://msdl.microsoft.com/download/symbols
(gdb) core hello3.dmp
[New Thread 0x1a80]
[New Thread 0x2e0]
[New Thread 0xd6c]
[New Thread 0x4e0]
Core was generated by `C:\cygwin64\home\Administrator\case10\hello\hello.exe'.
#0  0x00007ffcfbf002b4 in ntdll!ZwWaitForMultipleObjects () from C:\src\dlls-syms\ntdll.dll\ED90036F1ed000\ntdll.dll
[Current thread is 1 (Thread 0x1a80)]
(gdb) bt
#0  0x00007ffcfbf002b4 in ntdll!ZwWaitForMultipleObjects () from C:\src\dlls-syms\ntdll.dll\ED90036F1ed000\ntdll.dll
#1  0x00007ffcf80dcc7e in WaitForMultipleObjectsEx () from C:\src\dlls-syms\KERNELBASE.dll\7416F3AE293000\KERNELBASE.dll
#2  0x00007ffcf80dcb6e in WaitForMultipleObjects () from C:\src\dlls-syms\KERNELBASE.dll\7416F3AE293000\KERNELBASE.dll
#3  0x00007ffced165fe0 in cygwait (object=object@entry=0x0, timeout=timeout@entry=0x7ffffcb68, mask=mask@entry=11) at /usr/src/debug/cygwin-3.5.1-1/winsup/cygwin/cygwait.cc:75
#4  0x00007ffced1d6feb in clock_nanosleep (clk_id=clk_id@entry=1, flags=flags@entry=0, rqtp=rqtp@entry=0x7ffffcbf0, rmtp=rmtp@entry=0x7ffffcc00) at /usr/src/debug/cygwin-3.5.1-1/winsup/cygwin/signal.cc:130
#5  0x00007ffced1d73c9 in sleep (seconds=<optimized out>) at /usr/src/debug/cygwin-3.5.1-1/winsup/cygwin/signal.cc:174
#6  0x00007ffced2aa1cb in _sigfe () at sigfe.s:36
#7  0x00000001004010ad in main (argc=1, argv=0xa00000410) at hello.c:8

I am still compiling with the patch ... but if you can share your binary or even make a release that would help a lot (if you would include the cygwin patch 0001 in your branch .. don't know if this is the right thing to do because it probably eventually breaks the cygwin build process..).

I will definitely include patch 0001 in my builds from now on, it's cygwin specific anyways (I wonder why it isn't in upstream gdb).

[johannesthoma]

Thanks a lot for the fix it seems to work now with the cygwin patch (I couldn't get it to compile but this is some other problem, somehow my Windows build system crashed). Do you cross compile? (from Linux to Windows).

I will wait for a binary release then ... or retry tomorrow.

[ssbssa]

You can try the attached gdb.exe (it's the x86_64-python version). gdb.zip

Do you cross compile? (from Linux to Windows).

No, I build locally with MSYS2.

I will wait for a binary release then

This weekend the gdb 14.2 version should be released, I plan to create a binary release soon after.

[ssbssa]

I will wait for a binary release then ... or retry tomorrow.

The new 14.2 build contains the cygwin patch.

[johannesthoma]

Hi,

With the new build (gdb-14.2.90.20240303-x86_64) the stack dump still looks like this:

(gdb) bt                                                                                                
#0  0x00007fffe6cb02b4 in ntdll!ZwWaitForMultipleObjects () from C:\Windows\System32\ntdll.dll          
#1  0x00007fffe2edcc7e in WaitForMultipleObjectsEx () from C:\Windows\System32\KERNELBASE.dll           
#2  0x00007fffe2edcb6e in WaitForMultipleObjects () from C:\Windows\System32\KERNELBASE.dll             
#3  0x00007fffd7e25fe0 in cygwin1!.assert () from C:\cygwin64\bin\cygwin1.dll                           
#4  0x00007fffd7e96feb in sched_getscheduler () from C:\cygwin64\bin\cygwin1.dll                        
#5  0x00007fffd7e973c9 in sched_getscheduler () from C:\cygwin64\bin\cygwin1.dll                        
#6  0x00007fffd7f6a1cb in timegm () from C:\cygwin64\bin\cygwin1.dll                                    
#7  0x00000007ffffcd30 in ?? ()                                                                         
#8  0x0000000a00000420 in ?? ()                                                                         
#9  0x00000007ffffcdf0 in ?? ()                                                                         
#10 0x0000000000000000 in ?? ()                                                                         
(gdb)                                                                                                   

I am sure I am missing something obvious. Maybe a wrong cygwin1.dll?

Best regards,

• Johannes

[johannesthoma]

I used the non-python version.

[ssbssa]

Looks like you don't have the debuginfo for cygwin1.dll. In my tests even the cygwin-gdb needed them to get a proper backtrace.

[ssbssa]

I used the non-python version.

Just for my curiosity, why do you use the non-python version?

[johannesthoma]

Hi, I started using gdb in 1994 at that time there was no python support. Also I recall that I had a version dependency missing in python (I think when building or so) and to keep things simple I use the non-python version. Apart from that no particular reason. Its just that I am a little bit old-fashioned when it comes to software (like supporting ReactOS / Windows Server 2003 as WinDRBD target platform :) ).

[johannesthoma]

Btw after copying the cygwin.dll.dbg into the right path:

cp /usr/lib/debug/usr/bin/cygwin1.dll.dbg /bin/

after installing cygwin-debuginfo the stacks are displayed correctly. Will try to analyze the drbdadm dump now...

[johannesthoma]

Ok it looks like the machine where the dump of drbdadm was created uses other versions of the Windows system DLLs. I would expect the backtrace to contain mainly garbage if the NTDLL.DLL and others are from another Windows patch level, right? (then this isn't gdb's fault).

[ssbssa]

I'm a bit confused now, is drbdadm a cygwin program? It sounds like it's related to the WinDRBD stuff from #8.

If it's a cygwin program dump, then did you see what I wrote here about automatically getting windows DLLs from the microsoft symbol server?

[johannesthoma]

Ah yes drbdadm is a WinDRBD program linked against cygwin. Thank you for the symbol server hint, I will try that.

[johannesthoma]

Hi I copied the sysrv.dll from the developer EWDK ISO to the current working directory and tried to load the dump: output is:

warning: Can't find 'C:\Windows\System32\ntdll.dll' version 6.3.14393.6343.                                       
warning: Can't find 'C:\Windows\System32\kernel32.dll' version 6.3.14393.5786.                                    
warning: Can't find 'C:\Windows\System32\KERNELBASE.dll' version 6.3.14393.6343.                                  
warning: Can't find 'C:\Windows\System32\advapi32.dll' version 6.3.14393.6611.                                    
warning: Can't find 'C:\Windows\System32\msvcrt.dll' version 7.0.14393.2457.                                      
warning: Can't find 'C:\Windows\System32\sechost.dll' version 6.3.14393.6611.                                     
...

Any idea what might be wrong? If it just does not work that is also fine with me ... it is not a mission critical bug we are hunting :)

[ssbssa]

It should work, but as I wrote, you probably need to copy all 3 dlls: dbghelp/dbgcore/symsrv And I always put them directly where gdb.exe is, loading from the local directory might be disabled (since this is considered a security issue).

[ssbssa]

If it just does not work that is also fine with me ... it is not a mission critical bug we are hunting :)

This bug maybe is not mission critical, but the next one might be. I consider debugging of minidumps in gdb almost useless if the symbol server stuff doesn't work, so I hope you get it working.

[johannesthoma]

Ok I copied the DLLs into the bin directory where gdb is started from (didn't work with the . dir, the trick seems to be that the DLLs are in the same directory where the gdb binary used is located, just in case someone besides ssassb and myself reads this).

Now I need the correct cygwin.dll (with debug info of course) which was 3.4.5 where the dump was created. We're almost there :) :) :)

[johannesthoma]

Hi,

This bug maybe is not mission critical, but the next one might be. I consider debugging of minidumps in gdb almost useless if the symbol server stuff doesn't work, so I hope you get it working.

Sorry, I was talking about the WinDRBD (drbdadm.exe) bug, not the gdb bug. And yes I got the symbol server stuff working. Thanks a lot for pointing me into the right direction.