kaaveland / eugene

Careful With that Lock, Eugene
MIT License
31 stars 0 forks source link

Fix flaky tests that exec processes #103

Closed kaaveland closed 2 weeks ago

kaaveland commented 2 weeks ago

There's some sort of race condition or concurrency issue going on with these.

If I run this:

while cargo test; do echo "Running again"; done

I eventually get to where one or more of the tests that run git or postgres will fail in this way:

---- git::tests::test_unstaged stdout ----
thread 'git::tests::test_unstaged' panicked at eugene/src/git.rs:333:14:
called `Result::unwrap()` on an `Err` value: Error { context: ["Failed to execute `git status --porcelain` in \"/var/folders/tj/f5x9mm4n1tdcrkfpl6bbn2840000gn/T/eugene-test-unstagedGFxnfo\": No such file or directory (os error 2)", "Check if \"/var/folders/tj/f5x9mm4n1tdcrkfpl6bbn2840000gn/T/eugene-test-unstagedGFxnfo\" contains unstaged"], inner: GitError }
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:1654:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:1077:23
   4: eugene::git::tests::test_unstaged
             at ./src/git.rs:332:17
   5: eugene::git::tests::test_unstaged::{{closure}}
             at ./src/git.rs:322:23
   6: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

My hunch is that we're not using the tempfile crate correctly and we're somehow dropping the tempdir before we're done with it, or that we're concurrently generating the same tempdir name from two different tests. 🤔

kaaveland commented 2 weeks ago

Maybe the OS is cleaning up the dir underneath us before we're done using it? That seems unlikely. 🤔 Having no luck debugging this yet.

kaaveland commented 2 weeks ago

I think I've fixed this by fsyncing the temp directories and retrying the git commands that would sometimes fail over in #102