Cmake crash

[dmik]

All 9.2.0 builds of STDCPP6.DLL cause cmake --version to crash like this:

D:>cmake --version
cmake version 3.10.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
Assertion failed: h->magic == _UM_MAGIC_HEAP, file D:/Users/dmik/rpmbuild/BUILD/libc-0.1.3/src/emx/src/lib/malloc/umalloc.c, line 19

Killed by SIGABRT
pid=0x4e2d ppid=0x4dc8 tid=0x0001 slot=0x0090 pri=0x0200 mc=0x0001 ps=0x0010
C:\USR\BIN\CMAKE.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

Doing set LIBC_BREAKPOINT_ABORT=1 to cause LIBC generate a full .TRP instead of issuing a meaningless message results into the following report (thanks Elbert): https://pastebin.com/h1GqQ649

[dmik]

I did some checks. First, some symbols in the above TRP are wrong (perhaps because of the wrong versions of -debuginfo packages installed). Installing correct packages gives this: 4ED9_01.TRP.txt.

This is the essential part:

   EBP     Address    Module     Obj:Offset    Nearest Public Symbol
 --------  ---------  --------  -------------  -----------------------
 Trap  ->  1DF64121   LIBCN0    0001:00064121  abort.c#25 __std_abort + 19 0001:00064108 (abort.obj)

  Offset Name                 Type                         Hex Value
  ────── ──────────────────── ──────────────────────────── ─────────
  -32    set                  0x201                        0
  -24    act                  0x205                        4D

 0062FDB8  1DF4F4BA   LIBCN0    0001:0004F4BA  assert.c#18 __assert + 42 0001:0004F478 (assert.obj)

  Offset Name                 Type                         Hex Value
  ────── ──────────────────── ──────────────────────────── ─────────
   8     string               8 bit unsigned               1DF0FF29
   12    fname                8 bit unsigned               1DF0FEE0
   16    line                 32 bit unsigned              13

 0062FDE8  1DF0FFA4   LIBCN0    0001:0000FFA4  umalloc.c#20 __umalloc + 3C 0001:0000FF68 (D:\Users\dmik\rpmbuild\BUILD\libc-0.1.3\src\emx\src\lib\malloc\umalloc.c)

  Offset Name                 Type                         Hex Value
  ────── ──────────────────── ──────────────────────────── ─────────
   8     h                    pointer to type 0x202        20030000
   12    size                 32 bit unsigned              18

 0062FE28  1DF1C927   LIBCN0    0001:0001C927  malloc.c#25 __std_malloc + 27 0001:0001C900 (D:\Users\dmik\rpmbuild\BUILD\libc-0.1.3\src\emx\src\lib\malloc\malloc.c)

  Offset Name                 Type                         Hex Value
  ────── ──────────────────── ──────────────────────────── ─────────
   8     size                 32 bit unsigned              18

 0062FE48  1D7F000C   STDCPP6   0001:0001000C  new_op.cc#50 __Znwj + 1C 0001:0000FFF0 (D:\Users\dmik\rpmbuild\BUILD\gcc-9.2.0\libstdc++-v3\libsupc++\new_op.cc)

  Offset Name                 Type                         Hex Value
  ────── ──────────────────── ──────────────────────────── ─────────
   8     sz                   32 bit unsigned              18

 0062FE68  0008E1FC   CMAKE     0001:0007E1FC  stl_tree.h#463 __ZN20cmDynamicLoaderCache11GetInstanceEv + 1C 0001:0007E1E0 (cmDynamicLoader.cxx.obj)

 0062FE88  0008E32D   CMAKE     0001:0007E32D  cmDynamicLoader.cxx#97 __ZN15cmDynamicLoader10FlushCacheEv + B 0001:0007E322 (cmDynamicLoader.cxx.obj)

 0062FEA8  00012DD0   CMAKE     0001:00002DD0  cmakemain.cxx#202 main + A2 0001:00002D2E (E:\rpmbuild\BUILD\cmake-os2-master\Source\cmakemain.cxx)

  Offset Name                 Type                         Hex Value
  ────── ──────────────────── ──────────────────────────── ─────────
   0     ac                   32 bit signed                62FF64
   4     av                   0x3EAC                       10027
  -32    args                 0x41A9                       1DF6548B
  -20    args                 0x484                        10
  -20    args                 0x484                        10
  -32    args                 0x41A9                       1DF6548B

What I see there is that CMAKE does some C++ which eventually calls operator new which in turn calls malloc which in turn attempts to use the default (high-mem) heap, finds out that this heap's header is corrupt and then bails out.

Given that many other C++ apps work with this new STDCPP6.DLL like a charm (including heavy Qt 4 apps such as Qt Creator), I more tend to think that there is some problem in cmake. It is necessary to debug cmake code to understand what it is trying to do there, exactly. A typical reason for heap corruption is writing beyond the bounds of allocated objects.

[dmik]

Just rebuilding cmake with GCC 9.2.0 makes the issue go away. This makes things much harder to debug as source level debugging is not an option here. It's definitely some ABI break between the old and the new C++ DLL, but it must be something not documented. May be a bug of some sort (may be, OS/2 specific, may be not). I.e. the bug is present in GCC4 and it doesn't cause any problems when the code using its C++ DLL is also built with GCC4. Then it was fixed in GCC9 but since cmake still uses GCC4 stuffs, it blows now. Or it may be the other way around. It's just a wild guess.

But as I said, debugging such a problem is hard (it's deep in some C++ container code) and given that cmake is the only failing case so far (and it gets fixed by a rebuild), we will not invest time in it right now. If we face it again, we will deal with it.

By "the only failing case" I mean that I tested a lot of other apps built with the old GCC4 (and even with GCC3) running across the new C++ DLL from GCC9, including such monsters as Qt Creator (from Qt4) and Firefox 45.9, both of which heavily use a lot of C++. Both projects work smoothly so far.