search

rust-cli / rexpect

.github/workflows/ci.yml
https://docs.rs/rexpect
MIT License
329 stars 58 forks source link

Sporadically failing CI testcases #104

Open souze opened 1 year ago

souze commented 1 year ago

We have some sporadically failing testcases, which blocks CI. Has been discussed in #101 among other PRs. But I'm opening an issue to keep the discussion in one place.

I'm able to reliably reproduce the failure on my machine like so:

I=1; while cargo nextest run test_read_line; do I=$(($I+1)); echo "PASS $I"; done; echo "FAIL $I"
    Starting 1 tests across 1 binaries (19 skipped)
        FAIL [   0.207s] rexpect session::tests::test_read_line

--- STDOUT:              rexpect session::tests::test_read_line ---

running 1 test
test session::tests::test_read_line ... FAILED

failures:

failures:
    session::tests::test_read_line

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 19 filtered out; finished in 0.21s

--- STDERR:              rexpect session::tests::test_read_line ---
thread 'session::tests::test_read_line' panicked at 'assertion failed: `(left == right)`
  left: `"hans"`,
 right: `"hanshans"`', src/session.rs:437:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

   Canceling due to test failure: 0 tests still running
------------
     Summary [   0.207s] 1 tests run: 0 passed, 1 failed, 19 skipped
        FAIL [   0.207s] rexpect session::tests::test_read_line
error: test run failed
FAIL 270

It failed after 270 runs of the test. I'm running Ubuntu 20.

souze commented 1 year ago

So, I removed the code that turn echo off:

https://github.com/rust-cli/rexpect/blob/52edda700d39edabc1a7abcfa2d45ddb14e629c0/src/process.rs#L121-L123

That results in test_bash and test_bash_control_chars consistently failing. But it seems to remove the flakyness from test_read_line.

Very odd to me. I guess bash and cat set some flags inside their own process that affects us differently?