ziglang / zig

General-purpose programming language and toolchain for maintaining robust, optimal, and reusable software.
https://ziglang.org
MIT License
34.85k stars 2.55k forks source link

Fuzz tests are not discovered by test runner when re-running with same `--seed` argument #21410

Open AdamGoertz opened 1 month ago

AdamGoertz commented 1 month ago

Zig Version

0.14.0-dev.1573+4d81e8ee9

Steps to Reproduce and Observed Behavior

When running zig build test --fuzz --seed <seed> multiple times, on the second and later runs, the test runner exits with an error

$ ../zig/build/release/bin/zig build test --fuzz --seed 100
info: web interface listening at http://127.0.0.1:38091/
info: hint: pass --port 38091 to use this same port next time
^C/1] Fuzzing
$ ../zig/build/release/bin/zig build test --fuzz --seed 100
error: no fuzz tests found
error: the following build command failed with exit code 1:
/home/adam/fuzz_test/.zig-cache/o/8faf9687350f461d281e202daf8c0a3b/build /home/adam/zig/build/release/bin/zig /home/adam/zig/lib /home/adam/fuzz_test /home/adam/fuzz_test/.zig-cache /home/adam/.cache/zig --seed 100 -Z9b313e175a6f1d65 test --fuzz

This does not occur when a different seed is used, or when --seed is not set.

Different seeds:

$ ../zig/build/release/bin/zig build test --fuzz --seed 102
info: web interface listening at http://127.0.0.1:32903/
info: hint: pass --port 32903 to use this same port next time
^C/1] Fuzzing
$ ../zig/build/release/bin/zig build test --fuzz --seed 103
info: web interface listening at http://127.0.0.1:45863/
info: hint: pass --port 45863 to use this same port next time
^C/1] Fuzzing

No --seed

$ ../zig/build/release/bin/zig build test --fuzz
info: web interface listening at http://127.0.0.1:45375/
info: hint: pass --port 45375 to use this same port next time
^C/1] Fuzzing
$ ../zig/build/release/bin/zig build test --fuzz
info: web interface listening at http://127.0.0.1:43537/
info: hint: pass --port 43537 to use this same port next time
^C/1] Fuzzing

Expected Behavior

Fuzz tests should run even when the same seed is used multiple times.

andrewrk commented 1 month ago

@Vexu thanks for helping triage. Can you let me pick the milestone for stuff that I'm working on? I'm doing a big fuzzing push this release cycle, so I would have liked to see this issue when filtering for "no:milestone".

andrewrk commented 1 month ago

This is effectively a duplicate of #20803. I'll leave it open for now and address it along with that one. I have this partially completed locally.