Right now the determinism tests are not particularly easy to debug. This is bad because in the case of rarely triggered determinism bugs we could get into a situation where we know from a CI run that there is a non-deterministic actors message but not be able to isolate which message or even which test it is coming from.
This is not a big problem if we hit the bug locally, we just have to diff the new test-vectors and the test-vectors checked in. But we could make things nicer and have make determinism-check output information on the diffs.
This is problem on CI because the generated test-vector artifacts are not stored anywhere. We should find a way to either store the output for some time / in the case of error, or, probably this is better, add debug output to CI logs.
Right now the determinism tests are not particularly easy to debug. This is bad because in the case of rarely triggered determinism bugs we could get into a situation where we know from a CI run that there is a non-deterministic actors message but not be able to isolate which message or even which test it is coming from.
This is not a big problem if we hit the bug locally, we just have to diff the new test-vectors and the test-vectors checked in. But we could make things nicer and have
make determinism-checkoutput information on the diffs.This is problem on CI because the generated test-vector artifacts are not stored anywhere. We should find a way to either store the output for some time / in the case of error, or, probably this is better, add debug output to CI logs.