madsim-rs / madsim

Magical Deterministic Simulator for distributed systems in Rust.
Apache License 2.0
615 stars 41 forks source link

Check the result of the function in `check_determinism` #203

Open tomasol opened 2 months ago

tomasol commented 2 months ago

This is just an idea / feature request, it's totally fine to close the issue if it does not spark any interest.

Currently check_determinism verifies that the randomness state after running the input function twice is the same. If the input function is truly deterministic, it must return the same result. Here is how I currently check that a function is deterministic:

#[test]
fn ulid_generation_should_be_deterministic() {
    let mut builder_a = madsim::runtime::Builder::from_env();
    builder_a.check = true;
    let mut builder_b = madsim::runtime::Builder::from_env(); // Builder: Clone would be useful
    builder_b.check = true;
    builder_b.seed = builder_a.seed;
    assert_eq!(
        builder_a.run(|| async { ulid::Ulid::new() }),
        builder_b.run(|| async { ulid::Ulid::new() })
    );
}

It would be nice to have less boilerplate, given that the function returns a value implementing Debug and PartialEq

wangrunji0408 commented 2 months ago

Thanks for sharing your idea!

Besides checking for determinism, we also want to locate the source as soon as possible when uncertainty occurs. Therefore, check_determinism records the entire sequence of random states, and it immediately panics if it finds inconsistencies during the second run. This allows developers to roughly locate the uncertainty source through the backtrace. In comparison, only comparing function return values might miss some cases and doesn't quickly reveal problems. Moreover, this method won't work if the user's function does not return a random number, which I believe is more common in practice.

However, it would be nice to add return value check to the current check_determinism.