tests/unit-tests/margo-init

[carns]

I'm seeing a lot of failures that look like this in origin/main:

Error: /home/carns/working/src/mochi/mochi-margo/build/../tests/unit-tests/margo-init.c:122: assertion failed: t2-t1 >= 0.5 (0.106156 >= 0.5)
Error: child killed by signal 6 (Aborted)

I believe the assertion is meant to confirm that finalize_and_wait() doesn't complete until pending RPCs are done. I'm not sure why I would see this problem now, though.

My spack environment looks like this:

> spack find               e: mochi-margo-dev
==> In environment mochi-margo-dev
==> Root specs
argobots@main  cflags="-fsanitize=address -fno-omit-frame-pointer -g -Wall" ldflags="-fsanitize=address" 
json-c  cflags="-fsanitize=address -fno-omit-frame-pointer -g -Wall" ldflags="-fsanitize=address" 
mercury@master  cflags="-fsanitize=address -fno-omit-frame-pointer -g -Wall" ldflags="-fsanitize=address" 

==> Installed packages
-- linux-ubuntu22.10-skylake / gcc@12.2.0 -----------------------
argobots@main    cmake@3.24.2      libtool@2.4.7    mercury@master
autoconf@2.71    json-c@0.16       libunwind@1.6.2  valgrind@3.18.1
automake@1.16.5  libfabric@1.17.0  m4@1.4.19
==> 11 installed packages

[carns]

I'll try an environment that uses release versions of argobots and mercury to make sure there isn't some interaction there.

[mdorier]

The only two ways I could see this happening are:

1. The RPC is not executed (could happen if margo_forward queues it, returns, but nothing yields to the progress loop so it reaches the finalize_and_wait before mercury gets to trigger the RPC and spawn a ULT). Solution: add ABT_thread_yield() after margo_forward.
2. The RPC completes before the t1 = ABT_get_wtime() line, so finalize_and_wait doesn't have to wait for the RPC to complete. I find this unlikely since margo_thread_sleep in the RPC should yield to other ULTs, in particular the main ULT.

[carns]

Confirmed that the failure only happens with mercury@master (and is still present with current @master). The argobots version is irrelevant.

1) seems plausible; perhaps something changed in mercury that altered the timing.

I'm not sure if ABT_thread_yield() would be sufficient to guarantee the timing we want in the permutations that use a dedicated progress thread. It will only yield to other ULTs eligible to run on the same ES (of which there may be none), right?

[carns]

The test case is a "self" rpc, though, we could possibly just synchronize between the RPC fn and the main fn before the former sleeps to make sure that it is underway before the latter tries to finalize?

[mdorier]

That's a good option, yes.

[carns]

Whoops. We mis-diagnosed the problem. The RPC handler ult was actually starting in time, but the margo_forward() takes a full 1 second, as if the disable_response() call didn't work.