search

mirage / alcotest

A lightweight and colourful test framework
ISC License
456 stars 80 forks source link

Record exception backtraces during test suite runs #317

Closed craigfe closed 3 years ago

craigfe commented 3 years ago

Fix https://github.com/mirage/alcotest/issues/315.

craigfe commented 3 years ago

The 4.12+domains CI is consistently failing to run the Lwt examples binary on this PR, failing with the following segfault:

Starting program: /home/craigfe/t/alcotest/_build/default/examples/lwt/test.exe
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Testing `LwtUtils'.
This run has ID `WQI2PGWL'.

  [OK]          basic               0   Plain.
  [OK]          basic               1   Lwt.
  [OK]          exceptions          0   Plain.
  [OK]          exceptions          1   Lwt toplevel.
  [OK]          exceptions          2   Lwt internal.
  [OK]          switches            0   Allocate resource.
  [OK]          switches            1   Check resource deallocated.

Full test results in `~/t/alcotest/_build/default/examples/lwt/_build/_tests/LwtUtils'.
Test Successful in 0.002s. 7 tests run.

Program received signal SIGSEGV, Segmentation fault.
caml_darken (v=0, ignored=<optimized out>, state=<optimized out>) at major_gc.c:760
760 in major_gc.c
(gdb) bt
#0  caml_darken (v=0, ignored=<optimized out>, state=<optimized out>) at major_gc.c:760
#1  0x0000555555749fd6 in caml_iterate_global_roots (rootlist=0x55555585ae60 <caml_global_roots_old>, rootlist=0x55555585ae60 <caml_global_roots_old>, fdata=0x0, f=0x5555557307a0 <caml_darken>) at globroots.c:222
#2  caml_scan_global_roots (f=f@entry=0x5555557307a0 <caml_darken>, fdata=fdata@entry=0x0) at globroots.c:233
#3  0x0000555555731257 in cycle_all_domains_callback (domain=domain@entry=0x7ffff7b80000, unused=unused@entry=0x0, participating_count=<optimized out>, participating=participating@entry=0x555555864c40 <stw_request+64>) at major_gc.c:1092
#4  0x0000555555752540 in caml_try_run_on_all_domains_with_spin_work (handler=handler@entry=0x555555731180 <cycle_all_domains_callback>, data=data@entry=0x0, leader_setup=leader_setup@entry=0x0,
    enter_spin_callback=enter_spin_callback@entry=0x0, enter_spin_data=enter_spin_data@entry=0x0) at domain.c:977
#5  0x000055555575261a in caml_try_run_on_all_domains (handler=handler@entry=0x555555731180 <cycle_all_domains_callback>, data=data@entry=0x0, leader_setup=leader_setup@entry=0x0) at domain.c:991
#6  0x0000555555731fae in major_collection_slice (howmuch=<optimized out>, participant_count=participant_count@entry=0, barrier_participants=barrier_participants@entry=0x0, mode=mode@entry=Slice_interruptible) at major_gc.c:1376
#7  0x0000555555732138 in caml_major_collection_slice (howmuch=howmuch@entry=-1) at major_gc.c:1395
#8  0x0000555555750db5 in caml_poll_gc_work () at domain.c:1041
#9  0x0000555555734844 in caml_alloc (tag=<optimized out>, wosize=<optimized out>) at alloc.c:43
#10 caml_alloc (wosize=<optimized out>, tag=<optimized out>) at alloc.c:33
#11 0x000055555573c9e9 in caml_ml_out_channels_list (unit=<optimized out>) at io.c:568
#12 <signal handler called>
#13 camlStdlib__flush_all_232 () at stdlib.ml:350
#14 0x00005555556cd966 in camlStdlib__exit_475 () at stdlib.ml:561
#15 0x000055555566aa66 in camlAlcotest_engine__Monad__fun_400 () at src/alcotest-engine/monad.ml:32
#16 0x00005555556b3ee8 in camlCmdliner_term__fun_186 () at cmdliner_term.ml:25
#17 0x00005555556b7b59 in camlCmdliner__run_418 () at cmdliner.ml:117
#18 0x00005555556b7e98 in camlCmdliner__term_eval_444 () at cmdliner.ml:147
#19 0x00005555556b8c22 in camlCmdliner__eval_choice_inner_1653 () at cmdliner.ml:265
#20 0x000055555567807a in camlAlcotest_engine__Cli__run_with_args$27_1097 () at src/alcotest-engine/cli.ml:108
#21 0x0000555555645765 in camlDune__exe__Test__entry () at examples/lwt/test.ml:82
#22 0x0000555555640756 in caml_program ()
#23 <signal handler called>
#24 0x000055555572e947 in caml_startup_common (argv=0x7fffffffd6c8, pooling=<optimized out>, pooling@entry=0) at startup_nat.c:133
#25 0x000055555572e978 in caml_startup_exn (argv=<optimized out>) at startup_nat.c:137
#26 caml_main (argv=<optimized out>) at startup_nat.c:142
#27 0x000055555563ff4c in main (argc=<optimized out>, argv=<optimized out>) at main.c:41

The segfault happens at the very end of running the binary, during flushing on the call to exit 0. Not sure what to make of this, but it seems unrelated to the PR. @ctk21: any thoughts?

craigfe commented 3 years ago

Rebased and fixed the expect tests to account for changes made in https://github.com/mirage/alcotest/pull/326.

craigfe commented 3 years ago

FTR, I tried to avoid duplicating snapshot outputs between the native and JS expect tests (here), but couldn't find a diff that behaves consistently on all three platforms. (Unfortunately, it seems there's no patdiff available for a modern version of OCaml on our Windows CI runner.) Just doing with the duplicated snapshots for now.

Merging now as the CI is happy.

hhugo commented 3 years ago

FTR, I tried to avoid duplicating snapshot outputs between the native and JS expect tests (here), but couldn't find a diff that behaves consistently on all three platforms. (Unfortunately, it seems there's no patdiff available for a modern version of OCaml on our Windows CI runner.) Just doing with the duplicated snapshots for now.

Merging now as the CI is happy.

The major downside is that it's now easier to introduce a change behaving differently in native and js

craigfe commented 3 years ago

Agreed. If I'd found a way to do reproducible diffing, I'd have gone with that instead. For now, it'll be necessary to run some diff tool manually when tests are added.