Build fails on `dune exec foo.exe&` with Dune 2.3 and 2.4 randomly fail (2.0, 2.1 and 2.2 work)

anton-trunov commented 4 years ago

Expected Behavior

Scilla builds succeed (make test_server target).

Actual Behavior

Builds fail randomly on Dune v2.3 and v2.4, often times with different messages, e.g.

$ make test_server
./scripts/libff.sh
Found libff.a, not building again
dune build --profile dev @install
dune build --profile dev tests/scilla_client.exe
ln -sr _build/default/tests/scilla_client.exe _build/install/default/bin/scilla-client
dune exec src/runners/scilla_server.exe &
ulimit -n 4096; dune exec tests/testsuite.exe -- -print-diff true -runner sequential \
  -server true \
    -only-test "all_tests:0:contract_tests:0:these_tests_must_SUCCEED"
Error: exception Sys_error("_build/default/.dune/.dune-keep: No such file or
directory")
Raised by primitive operation at file "stdlib.ml", line 324, characters 29-55
Called from file "src/stdune/io.ml", line 58, characters 17-37
Called from file "src/dune/context.ml", line 244, characters 2-61
Called from file "src/dune/context.ml", line 404, characters 14-75
Called from file "src/fiber/fiber.ml", line 196, characters 36-41
Called from file "src/fiber/fiber.ml", line 102, characters 8-15

or it may look like this

$ make test_server
./scripts/libff.sh
Found libff.a, not building again
dune build --profile dev @install
dune build --profile dev tests/scilla_client.exe
ln -sr _build/default/tests/scilla_client.exe _build/install/default/bin/scilla-client
dune exec src/runners/scilla_server.exe &
dune exec tests/testsuite.exe -- -print-diff true -runner sequential \
  -server true \
-only-test "all_tests:0:contract_tests:0:these_tests_must_SUCCEED"
Error: Error: unlink: _build/default/.dune/configurator: No such file or
directory
FFFFFFFFFFFFFFFFFFFFFFFF.FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF....FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
==============================================================================
Error: all_tests:0:contract_tests:0:these_tests_must_SUCCEED:39:listiter:3:listiter_1_disable_validate_json.

Notice that we run the scilla_server executable as a background process here (I don't know if that's relevant).

Reproduction

Our setup is a bit complicate, but one can check the results of Travis CI in this PR https://github.com/Zilliqa/scilla/pull/803. Or feel free to push test PRs. It builds just fine with Dune 2.0, 2.1, 2.2.

Specifications

Version of dune (output of dune --version): 2.3.1, 2.4.0
Version of ocaml (output of ocamlc --version): 4.07.1, 4.08.1, 4.09.1
Operating system (distribution and version): macOS 10.14.6, Ubuntu 16.04.6 LTS

ghost commented 4 years ago

Notice that we run the scilla_server executable as a background process here (I don't know if that's relevant).

That's the issue. There is a race condition between the two dune running. Ideally, dune would set a lock but we currently don't do that. Do you have an option in scilla_server to "daemonize"? This way you wouldn't have to rely on &, i.e. you'd do:

dune exec src/runners/scilla_server.exe --daemonise
ulimit ...

Otherwise, you'll have to do this:

dune build src/runners/scilla_server.exe
./_build/default/src/runners/scilla_server.exe &

You can also add (mode promote) to the executable stanza so that dune copies the executable in the source tree.

anton-trunov commented 4 years ago

@diml Thank you so much for your quick reply! And for the promote suggestion too, this looks very handy.

rgrinberg commented 2 years ago

Duplicate of #236

ocaml / dune