Reworking of lists, etc

[viega]

In order to support more natural list ops, I made the list implementation use a read/write lock, and relegated the lock-free array to the cobwebs. I fixed some little bugs around it all.

I also refactored the collection algorithm to be breadth first instead of depth first, and re-arranged most structs to put pointers first, then 64-bit ints, since there were definitely some unaligned pointer issues w/ GCC (which could possibly persist in the stack layout, but hopefully not).

Debug builds now automatically run asan and ubsan; currently on my machine, there is one UB error, but as far as I can tell it is some false positive. What it's saying doesn't seem to map to the code even a little bit.

Particularly:

1. It's complaining about a bad boolean load.
2. The boolean is local, and explicitly initialized with bool flush = false.
3. The debugger agrees that the keyword argument that could modifies the address does not, and that, indeed, nothing modifies it on the call in question, when breaking on the function (verified w/ a hardware watchpoint).
4. This happens one and only one time across thousands and thousands of similar calls. But always in the same place.
5. HOWEVER, when the stack frame is entered, the value is somehow true already.
6. When I rearrange the declarations in the stack frame, I get the thing still.

I think there's a good chance there's just some weird ASAN byte being written, and it's just some weird false. But maybe not. Either way, I didn't chase it for long, and don't plan to (spent more time describing it than working it).

Closes #59 Closes #55 Closes #57 Closes #61 Closes #42

[ee7]

Just in case we didn't see, the tests crash in CI for Linux:

./dev: line 134:  4792 Segmentation fault      (core dumped) ./c4test $@

It looks like when c4test produces SIGSEGV, the dev run command still exits with 0, causing the CI job to indicate success. I've reopened #42 to track that.

On my machine: all new tests pass, but basic23.c4m crashes. Valgrind:

$ valgrind --leak-check=no --track-origins=yes -s debug/c4test tests/basic23.c4m
==105310== Memcheck, a memory error detector
==105310== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==105310== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
==105310== Command: debug/c4test tests/basic23.c4m
==105310== 
==105310== Warning: set address range perms: large range [0x59cb8000, 0xd9cba000) (defined)
==105310== Invalid read of size 8
==105310==    at 0x12A944: c4m_repr_exception_stack_no_vm (exceptions.c:143)
==105310==    by 0x12AAFC: c4m_exception_uncaught (exceptions.c:168)
==105310==    by 0x12AB56: c4m_exception_raise (exceptions.c:180)
==105310==    by 0x133D33: c4m_stream_init (streams.c:189)
==105310==    by 0x11E7F1: _c4m_new (object.c:471)
==105310==    by 0x1344B0: c4m_init_std_streams (streams.c:734)
==105310==    by 0x116CF5: c4m_init (init.c:41)
==105310==    by 0x4EEFDC3: call_init (libc-start.c:145)
==105310==    by 0x4EEFDC3: __libc_start_main@@GLIBC_2.34 (libc-start.c:347)
==105310==    by 0x116DF4: (below main) (in /foo/libcon4m/debug/c4test)
==105310==  Address 0x8 is not stack'd, malloc'd or (recently) free'd

And in case any are important: note that this PR also currently makes the Linux + GCC 14 CI job produce quite a few warnings (scroll down from here) that don't appear on main:

• -Wmaybe-uninitialized
• -Warray-bounds=
• -Wclobbered
• -Wstrict-aliasing
• -Wimplicit-fallthrough=
• -Wunused-result
• -Wunused-but-set-variable

By the way: if it's better/faster overall, I don't mind us adding test cases that currently fail, or merging PRs where CI indicates failure. With #38 it was never my intention to insist right now that all tests pass in CI.

[viega]

Yes this is why it is in draft.

On Tue, Jul 2, 2024 at 6:29 AM ee7 @.***> wrote:

Just in case we didn't see, the tests crash in CI for Linux:

./dev: line 134: 4792 Segmentation fault (core dumped) ./c4test $@

The dev script is still broken, because the job indicates success.

On my machine: all new tests pass, but basic23.c4m crashes. Valgrind:

$ valgrind --leak-check=no --track-origins=yes -s debug/c4test tests/basic23.c4m==105310== Memcheck, a memory error detector==105310== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.==105310== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info==105310== Command: debug/c4test tests/basic23.c4m==105310== ==105310== Warning: set address range perms: large range [0x59cb8000, 0xd9cba000) (defined)==105310== Invalid read of size 8==105310== at 0x12A944: c4m_repr_exception_stack_no_vm (exceptions.c:143)==105310== by 0x12AAFC: c4m_exception_uncaught (exceptions.c:168)==105310== by 0x12AB56: c4m_exception_raise (exceptions.c:180)==105310== by 0x133D33: c4m_stream_init (streams.c:189)==105310== by 0x11E7F1: _c4m_new (object.c:471)==105310== by 0x1344B0: c4m_init_std_streams (streams.c:734)==105310== by 0x116CF5: c4m_init (init.c:41)==105310== by 0x4EEFDC3: call_init (libc-start.c:145)==105310== by 0x4EEFDC3: __libc_start_main@@GLIBC_2.34 (libc-start.c:347)==105310== by 0x116DF4: (below main) (in /foo/libcon4m/debug/c4test)==105310== Address 0x8 is not stack'd, malloc'd or (recently) free'd

And in case any are important: note that this PR also currently makes the Linux + GCC 14 CI job produce quite a few warnings (scroll down from here https://github.com/crashappsec/libcon4m/actions/runs/9754378238/job/26921197654#step:7:89) that don't appear on main:

• -Wmaybe-uninitialized
• -Warray-bounds=
• -Wclobbered
• -Wstrict-aliasing
• -Wimplicit-fallthrough=
• -Wunused-result
• -Wunused-but-set-variable

— Reply to this email directly, view it on GitHub https://github.com/crashappsec/libcon4m/pull/60#issuecomment-2202680046, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABELGQL2JXOBT4S7VOIUW7DZKJ6ILAVCNFSM6AAAAABKGVAISOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBSGY4DAMBUGY . You are receiving this because you authored the thread.Message ID: @.***>

[viega]

@ee7 Tests are working on my x86 linux VM with this PR as-is. No clue what the crash here is, and no diagnostics.

[ee7]

basic23.c4m now passes for me locally on x86_64 Linux with gcc and clang. ASan and valgrind still produce the same complaints from https://github.com/crashappsec/libcon4m/issues/46 though.

I'll dig a bit more, but it's definitely nothing related to the remaining enabled-by-default GCC warnings introduced by this PR, right?

With https://github.com/crashappsec/libcon4m/pull/60/commits/c55983c47b0fdba8eda27f456327b2f46f60c3fb:

In file included from ../include/con4m.h:101,
                 from ../src/con4m/compiler/compile.c:2:
../src/con4m/compiler/compile.c: In function ‘c4m_initial_load_one’:
../include/con4m/exception.h:75:31: warning: variable ‘_c4x_current_exception’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
   75 |     volatile c4m_exception_t *_c4x_current_exception = NULL;             \
      |                               ^~~~~~~~~~~~~~~~~~~~~~
../src/con4m/compiler/compile.c:619:5: note: in expansion of macro ‘C4M_TRY’
  619 |     C4M_TRY
      |     ^~~~~~~
[57/158] Compiling C object libcon4m.a.p/src_con4m_vm.c.o
../src/con4m/vm.c: In function ‘c4m_vm_runloop’:
../src/con4m/vm.c:146:23: warning: array subscript [-2147352574, 131072] is outside array bounds of ‘c4m_stack_value_t[131072]’ [-Warray-bounds=]
  146 |     if (tstate->sp >= &tstate->stack[STACK_SIZE - (_n) + 1]) { \
      |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/con4m/vm.c:864:21: note: in expansion of macro ‘STACK_REQUIRE_VALUES’
  864 |                     STACK_REQUIRE_VALUES(i->arg);
      |                     ^~~~~~~~~~~~~~~~~~~~
In file included from ../include/con4m/datatypes.h:34,
                 from ../include/con4m/base.h:71,
                 from ../include/con4m.h:56,
                 from ../src/con4m/vm.c:2:
../include/con4m/datatypes/vm.h:583:23: note: while referencing ‘stack’
  583 |     c4m_stack_value_t stack[STACK_SIZE];
      |                       ^~~~~
In file included from ../include/con4m.h:101:
../include/con4m/exception.h:75:31: warning: variable ‘_c4x_current_exception’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
   75 |     volatile c4m_exception_t *_c4x_current_exception = NULL;             \
      |                               ^~~~~~~~~~~~~~~~~~~~~~
../src/con4m/vm.c:805:5: note: in expansion of macro ‘C4M_TRY’
  805 |     C4M_TRY
      |     ^~~~~~~
[62/158] Compiling C object libcon4m.a.p/src_con4m_compiler_parse.c.o
../src/con4m/compiler/parse.c: In function ‘extern_block’:
../src/con4m/compiler/parse.c:1270:18: warning: variable ‘got_box’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
 1270 |     bool         got_box      = false;
      |                  ^~~~~~~
../src/con4m/compiler/parse.c:1271:18: warning: variable ‘got_pure’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
 1271 |     bool         got_pure     = false;
      |                  ^~~~~~~~
../src/con4m/compiler/parse.c:1272:18: warning: variable ‘got_holds’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
 1272 |     bool         got_holds    = false;
      |                  ^~~~~~~~~
../src/con4m/compiler/parse.c:1273:18: warning: variable ‘got_allocs’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
 1273 |     bool         got_allocs   = false;
      |                  ^~~~~~~~~~

[ee7]

Linux + Clang doesn't crash in CI (see https://github.com/crashappsec/libcon4m/pull/62, based on top of c55983c47b0fdba8eda27f456327b2f46f60c3fb), which may or may not be a clue.

[ee7]

On top of 714cb38fb3b8084c1eeed9468dbb728b04db0d2e, ASan in the Linux CI environment:

==3972==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f296740bea0 at pc 0x563d570c1437 bp 0x7fff3e6a4b10 sp 0x7fff3e6a4b00
READ of size 8 at 0x7f296740bea0 thread T0
    #0 0x563d570c1436 in scan_range_for_allocs ../src/con4m/collect.c:453
    #1 0x563d570c1b5f in scan_roots ../src/con4m/collect.c:476
    #2 0x563d570c1b5f in raw_trace ../src/con4m/collect.c:523
    #3 0x563d570c1b5f in c4m_collect_arena ../src/con4m/collect.c:586
    #4 0x563d570b48ee in c4m_alloc_from_arena ../src/con4m/gcbase.c:573
    #5 0x563d570ccd8d in _c4m_gc_raw_alloc ../src/con4m/gcbase.c:339
    #6 0x563d570ccd8d in _c4m_new ../src/con4m/object.c:432
    #7 0x563d570b7a3d in c4m_to_utf32 ../src/con4m/string.c:481
    #8 0x563d57113157 in c4m_str_vformat ../src/con4m/string.c:471
    #9 0x563d5710edbf in _c4m_cstr_format ../src/con4m/format.c:610
    #10 0x563d57117418 in internal_normalize_and_join ../src/con4m/path.c:109
    #11 0x563d571d390e in get_file_compile_ctx.isra.0 ../src/con4m/compiler/compile.c:198
    #12 0x563d5711ca65 in ctx_init_from_file_uri ../src/con4m/compiler/compile.c:476
    #13 0x563d5711ca65 in c4m_init_module_from_loc ../src/con4m/compiler/compile.c:495
    #14 0x563d5711ca65 in c4m_new_compile_context ../src/con4m/compiler/compile.c:590
    #15 0x563d57130b6c in c4m_compile_from_entry_point ../src/con4m/compiler/compile.c:1050
    #16 0x563d570a78a1 in test_compiler ../src/tests/test.c:429
    #17 0x563d570a78a1 in main ../src/tests/test.c:519
    #18 0x7f296962a1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3)
    #19 0x7f296962a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3)
    #20 0x563d570aefd4 in _start (/home/runner/work/libcon4m/libcon4m/build/c4test+0x51fd4) (BuildId: 3f730bc3f72ef638b61013b72a7a55d583aa9118)

Address 0x7f296740bea0 is located in stack of thread T0 at offset 32 in frame
    #0 0x563d570c0cef in forward_one_allocation ../src/con4m/collect.c:365

  This frame has 1 object(s):
    [32, 40) 'new_arena' (line 364) <== Memory access at offset 32 is inside this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-return ../src/con4m/collect.c:453 in scan_range_for_allocs
Shadow bytes around the buggy address:
  0x7f296740bc00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7f296740bc80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7f296740bd00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7f296740bd80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7f296740be00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
=>0x7f296740be80: f5 f5 f5 f5[f5]f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7f296740bf00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7f296740bf80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7f296740c000: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7f296740c080: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7f296740c100: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3972==ABORTING
[-- libcon4m --] Exit status: 1

[viega]

@ee7 No, I'm running w/ GCC on my x86 linux instance and it is just fine (gcc 14). I don't have clang installed on it.

[viega]

@orangematt At this point it's definitely working fine on linux w/ GCC on a cloud instance running a current gcc; the gcc runner crashes w/o diagnostics but I'm ready to move on. Happy to only use clang in the runners.

[viega]

This is a false positive; the garbage collector needs to scan the stack for pointers, and asan puts guards all over the stack to detect when programs read stuff on the stack they didn't write.

On Tue, Jul 2, 2024 at 12:18 PM ee7 @.***> wrote:

On top of 714cb38 https://github.com/crashappsec/libcon4m/commit/714cb38fb3b8084c1eeed9468dbb728b04db0d2e, ASan in CI:

==3972==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f296740bea0 at pc 0x563d570c1437 bp 0x7fff3e6a4b10 sp 0x7fff3e6a4b00 READ of size 8 at 0x7f296740bea0 thread T0

0 0x563d570c1436 in scan_range_for_allocs ../src/con4m/collect.c:453

#1 0x563d570c1b5f in scan_roots ../src/con4m/collect.c:476
#2 0x563d570c1b5f in raw_trace ../src/con4m/collect.c:523
#3 0x563d570c1b5f in c4m_collect_arena ../src/con4m/collect.c:586
#4 0x563d570b48ee in c4m_alloc_from_arena ../src/con4m/gcbase.c:573
#5 0x563d570ccd8d in _c4m_gc_raw_alloc ../src/con4m/gcbase.c:339
#6 0x563d570ccd8d in _c4m_new ../src/con4m/object.c:432
#7 0x563d570b7a3d in c4m_to_utf32 ../src/con4m/string.c:481
#8 0x563d57113157 in c4m_str_vformat ../src/con4m/string.c:471
#9 0x563d5710edbf in _c4m_cstr_format ../src/con4m/format.c:610
#10 0x563d57117418 in internal_normalize_and_join ../src/con4m/path.c:109
#11 0x563d571d390e in get_file_compile_ctx.isra.0 ../src/con4m/compiler/compile.c:198
#12 0x563d5711ca65 in ctx_init_from_file_uri ../src/con4m/compiler/compile.c:476
#13 0x563d5711ca65 in c4m_init_module_from_loc ../src/con4m/compiler/compile.c:495
#14 0x563d5711ca65 in c4m_new_compile_context ../src/con4m/compiler/compile.c:590
#15 0x563d57130b6c in c4m_compile_from_entry_point ../src/con4m/compiler/compile.c:1050
#16 0x563d570a78a1 in test_compiler ../src/tests/test.c:429
#17 0x563d570a78a1 in main ../src/tests/test.c:519
#18 0x7f296962a1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3)
#19 0x7f296962a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3)
#20 0x563d570aefd4 in _start (/home/runner/work/libcon4m/libcon4m/build/c4test+0x51fd4) (BuildId: 3f730bc3f72ef638b61013b72a7a55d583aa9118)

Address 0x7f296740bea0 is located in stack of thread T0 at offset 32 in frame

0 0x563d570c0cef in forward_one_allocation ../src/con4m/collect.c:365

This frame has 1 object(s): [32, 40) 'new_arena' (line 364) <== Memory access at offset 32 is inside this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions are supported) SUMMARY: AddressSanitizer: stack-use-after-return ../src/con4m/collect.c:453 in scan_range_for_allocs Shadow bytes around the buggy address: 0x7f296740bc00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x7f296740bc80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x7f296740bd00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x7f296740bd80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x7f296740be00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 =>0x7f296740be80: f5 f5 f5 f5[f5]f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x7f296740bf00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x7f296740bf80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x7f296740c000: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x7f296740c080: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x7f296740c100: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==3972==ABORTING [-- libcon4m --] Exit status: 1

— Reply to this email directly, view it on GitHub https://github.com/crashappsec/libcon4m/pull/60#issuecomment-2203740537, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABELGQMIDEKHUT4U66KDEU3ZKLHETAVCNFSM6AAAAABKGVAISOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBTG42DANJTG4 . You are receiving this because you authored the thread.Message ID: @.***>

[viega]

No, this has nothing to do with anything. One of the complaints is talking about a pointer being out of bounds that is supposed to be PAST the end (it isn't dereferenced, it is used to test for stack underflow). That pointer is never dereferenced, period, so it's not a problem. That one I could easily make go away, but pointless.

The rest are the same issue, and fall in the land of UB behavior that is consistent across the platforms we use at least. It's saying an exception being thrown might wipe out those variables. If so, that would put everything into an infinite loop and not crash (it would generally not consume any more memory).

That one requires a lot more refactoring just to get rid of the warning; you cannot move them into the context object directly (or into tls), since the functions can be called recursively. Basically one would have to get a reference from the caller or use some dynamic memory.

No point in overcomplicating that until I'm ready to think about what error handling should end up looking like w/in the language.

basic23.c4m now passes for me locally on x86_64 Linux with gcc and clang. ASan and valgrind still produce the same complaints from #46 though.

I'll dig a bit more, but it's definitely nothing related to the remaining enabled-by-default GCC warnings introduced by this PR, right?

With c55983c:

In file included from ../include/con4m.h:101,
                 from ../src/con4m/compiler/compile.c:2:
../src/con4m/compiler/compile.c: In function ‘c4m_initial_load_one’:
../include/con4m/exception.h:75:31: warning: variable ‘_c4x_current_exception’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
   75 |     volatile c4m_exception_t *_c4x_current_exception = NULL;             \
      |                               ^~~~~~~~~~~~~~~~~~~~~~
../src/con4m/compiler/compile.c:619:5: note: in expansion of macro ‘C4M_TRY’
  619 |     C4M_TRY
      |     ^~~~~~~
[57/158] Compiling C object libcon4m.a.p/src_con4m_vm.c.o
../src/con4m/vm.c: In function ‘c4m_vm_runloop’:
../src/con4m/vm.c:146:23: warning: array subscript [-2147352574, 131072] is outside array bounds of ‘c4m_stack_value_t[131072]’ [-Warray-bounds=]
  146 |     if (tstate->sp >= &tstate->stack[STACK_SIZE - (_n) + 1]) { \
      |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/con4m/vm.c:864:21: note: in expansion of macro ‘STACK_REQUIRE_VALUES’
  864 |                     STACK_REQUIRE_VALUES(i->arg);
      |                     ^~~~~~~~~~~~~~~~~~~~
In file included from ../include/con4m/datatypes.h:34,
                 from ../include/con4m/base.h:71,
                 from ../include/con4m.h:56,
                 from ../src/con4m/vm.c:2:
../include/con4m/datatypes/vm.h:583:23: note: while referencing ‘stack’
  583 |     c4m_stack_value_t stack[STACK_SIZE];
      |                       ^~~~~
In file included from ../include/con4m.h:101:
../include/con4m/exception.h:75:31: warning: variable ‘_c4x_current_exception’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
   75 |     volatile c4m_exception_t *_c4x_current_exception = NULL;             \
      |                               ^~~~~~~~~~~~~~~~~~~~~~
../src/con4m/vm.c:805:5: note: in expansion of macro ‘C4M_TRY’
  805 |     C4M_TRY
      |     ^~~~~~~
[62/158] Compiling C object libcon4m.a.p/src_con4m_compiler_parse.c.o
../src/con4m/compiler/parse.c: In function ‘extern_block’:
../src/con4m/compiler/parse.c:1270:18: warning: variable ‘got_box’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
 1270 |     bool         got_box      = false;
      |                  ^~~~~~~
../src/con4m/compiler/parse.c:1271:18: warning: variable ‘got_pure’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
 1271 |     bool         got_pure     = false;
      |                  ^~~~~~~~
../src/con4m/compiler/parse.c:1272:18: warning: variable ‘got_holds’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
 1272 |     bool         got_holds    = false;
      |                  ^~~~~~~~~
../src/con4m/compiler/parse.c:1273:18: warning: variable ‘got_allocs’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]
 1273 |     bool         got_allocs   = false;
      |                  ^~~~~~~~~~

[ee7]

@ee7 No, I'm running w/ GCC on my x86 linux instance and it is just fine (gcc 14). I don't have clang installed on it.

OK. But just in case it was unclear: the ASan output in https://github.com/crashappsec/libcon4m/pull/60#issuecomment-2203740537 was from the CI setup in this PR, with Linux + GCC (not clang), modified just to add ASan.

[ee7]

the gcc runner crashes w/o diagnostics

In case it helps: it looks like the crash in CI is the same as what I used to see in this PR locally for basic23.c4m. But it crashes even for just basic1.c4m.

On top of 714cb38fb3b8084c1eeed9468dbb728b04db0d2e, disabling FLTO (for a better trace) and then running valgrind in the Linux + gcc CI environment:

$ valgrind --leak-check=no --track-origins=yes ./build/c4test tests
Memcheck, a memory error detector
Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
Command: ./build/c4test tests

Warning: set address range perms: large range [0x5590000, 0x15592000) (defined)
Invalid read of size 8
   at 0x12BCCB: c4m_repr_exception_stack_no_vm (exceptions.c:143)
   by 0x12BE90: c4m_exception_uncaught (exceptions.c:168)
   by 0x12BEF3: c4m_exception_raise (exceptions.c:180)
   by 0x1357CB: c4m_stream_init (streams.c:189)
   by 0x11F2D6: _c4m_new (object.c:471)
   by 0x135FDA: c4m_init_std_streams (streams.c:734)
   by 0x1172E2: c4m_init (init.c:41)
   by 0x4EA2303: call_init (libc-start.c:145)
   by 0x4EA2303: __libc_start_main@@GLIBC_2.34 (libc-start.c:347)
   by 0x1173E4: (below main) (in /home/runner/work/libcon4m/libcon4m/build/c4test)
 Address 0x8 is not stack'd, malloc'd or (recently) free'd

Process terminating with default action of signal 11 (SIGSEGV)
[...]
 If you believe this happened as a result of a stack
 overflow in your program's main thread (unlikely but
 possible), you can try to increase the size of the
 main thread stack using the --main-stacksize= flag.
 The main thread stack size used in this run was 16777216.

HEAP SUMMARY:
    in use at exit: 99,482 bytes in 3,105 blocks
  total heap usage: 4,914 allocs, 1,809 frees, 200,016 bytes allocated

For a detailed leak analysis, rerun with: --leak-check=full

For lists of detected and suppressed errors, rerun with: -s
ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

[viega]

Nothing in that trace has changed, perhaps since you started with the runner.

To get that error, there must be no file descriptor for stdout. Stdin opens, and, in the trace it's croaking on, it basically take the raw value of 'stdout', casts it to an int64, and passes it to something that checks to see if it's 0 or greater.

That's throwing an exception because it's got nothing to open, but since you have no stdout, it's crashing.

[viega]

@ee7 Aha, my spider sense was correct; older GCC made different alignment choices (the GC requires aligned pointers). Tests seem to mostly work both places; one regression issue that I'm about to look at.

Note that x86 does not have an architectural requirement for pointer alignment, so the stack might be an issue, and it's too much work to go through and ensure that. There's a compile-time paranoid option for stack scanning now, that might help if we have problems. Definitely don't ever run it on ARM; attempting to load an unaligned pointer is a crasher.

[viega]

@orangematt I'm done tweaking the thing, this is finally ready.