search

DynamicDevices / meta-mono

A Yocto / OpenEmbedded layer providing Mono and dotNet support
MIT License
15 stars 27 forks source link

mono-native 6.12.0.161 build error for Fedora 38 host #180

Closed zboszor closed 1 year ago

zboszor commented 1 year ago

Not sure what causes this. Kernel 6.2.x? GLIBC? GCC 13?

... [C parts of Mono build up to this point]
CSC     [build-linux] gensources.exe
Makefile:20: warning: overriding recipe for target 'test-local'
../../build/library.make:208: warning: ignoring old recipe for target 'test-local'
Makefile:20: warning: overriding recipe for target 'test-local'
../../build/library.make:208: warning: ignoring old recipe for target 'test-local'

=================================================================
        Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
        Native stacktrace:
=================================================================
        0x4a3dd2 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x4a4179 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x455301 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x41ba1f - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x7fe2e73bcb70 - /lib64/libc.so.6 : 
        0x49fee8 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x45a67d - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x41d1a6 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x6143ef - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x616e8f - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x572d81 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x424c59 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x42af1c - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : mono_main
        0x41b1ff - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : 
        0x7fe2e73a6b4a - /lib64/libc.so.6 : 
        0x7fe2e73a6c0b - /lib64/libc.so.6 : __libc_start_main
        0x41b7d5 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.161-r0/mono-6.12.0.122/mono/mini/mono : _start

=================================================================
        Telemetry Dumper:
=================================================================
No threads attached to runtime.

=================================================================
        External Debugger Dump:
=================================================================
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

=================================================================
        Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x49fee8):0x49fed8  83 c2 01 48 83 fa 01 0f 87 a9 00 00 00 83 e8 05  ...H............
0x49fee8  c6 03 e8 89 43 01 4d 85 f6 74 4d 4c 89 6b 06 41  ....C.M..tML.k.A
0x49fef8  bc 0e 00 00 00 c6 43 05 08 83 fd 0e 75 4a 4d 85  ......C.....uJM.
0x49ff08  ff 74 03 41 89 2f 89 ee 48 89 df e8 58 c1 ff ff  .t.A./..H...X...

The same crash occurs a couple of times for different DLLs until the build finally stops.

FWIW, I looked at the Fedora SRPM / spec for possible fixes.

I am out of ideas at the moment.

@ajlennon Any idea?

ajlennon commented 1 year ago

How very odd. No I don't really know I am afraid. I only run Ubuntu myself.

Have you had a go at restricting parallel builds?

It's not related to memory usage is it?

zboszor commented 1 year ago

How very odd. No I don't really know I am afraid. I only run Ubuntu myself.

Have you had a go at restricting parallel builds?

No, because the mono recipe runs make directly, restricting parallel builds to -j1 out of the box.

The recipe should use oe_runmake to do parallel build. But using it doesn't influence anything regarding the observed crashes, only the speed to reproduce them.

It's not related to memory usage is it?

It doesn't seem so.

ajlennon commented 1 year ago

Most odd. Has anybody upstream had anything to say about this maybe?

ajlennon commented 1 year ago

This may not be a good idea but potentially we could try to replicate this by making a docker CI build container using Fedora rather than Ubuntu?

zboszor commented 1 year ago

This is indeed strange.

The crash dump (i.e. the bytes as shown in the Memory around native instruction pointer message is identical across different DLL builds but it's sporadic in the sense that sometimes different DLLs show the issue.

And it WAS working previously. So it's an upgrade that causes this.

(after a lot of trial and error today)

TADA: 6.2.12 kernel allows building mono-native, 6.2.13 and 6.2.14 started crashing the Mono build.

What's left is to

zboszor commented 1 year ago

Sadly, 6.3.1 also crashes the build.

zboszor commented 1 year ago

Reported the issue in the Fedora bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2196250

ajlennon commented 1 year ago

Very very odd indeed eh

zboszor commented 1 year ago

FWIW, the cross-compiled Mono seems to work with kernel 6.2.14.