search

raiden-network / raiden

Raiden Network
https://developer.raiden.network
Other
1.84k stars 378 forks source link

Flaky tests #4193

Closed karlb closed 5 years ago

karlb commented 5 years ago

Many tests do not reliably succeed. I've compile the following list by installing pytest-repeat and running all tests five times in random order with

pytest -Wignore raiden/tests/ -n 20 --count 5 --random --show-capture=no

I marked all failed tests as flaky using https://github.com/box/flaky and retried until I got that command to run without a single (non-retried) error.

Since the number of flaky tests is quite high, I assume that there are some root causes that cause many tests to be flaky. Please split subsets flaky tests into separate issues whenever you identify a common root cause. Some tests might have failed due to increased load, but this is still something that should be fixed since the load was not overly high.

See also https://github.com/raiden-network/raiden/pull/4208.

LefterisJP commented 5 years ago

Hehe this issue is a deja vu: https://github.com/raiden-network/raiden/issues/389 Some tests got flaky again it seems as they were also in that issue and are also here, like the settlement tests.

Also related:

hackaugusto commented 5 years ago

Note: I believe most of these tests are not necessarily flaky when run in isolation. It seems a problem with executing the suite in parallel, which is not necessarily a good idea because of gevent.

karlb commented 5 years ago

Yes, there are different reasons for the observed flakiness, so far I observed:

I'll burn some CPU time for tests while working on other stuff so that I'll hopefully be able to assign specific reasons to specific tests. I do not believe that tests can't be run in parallel reliably at all due to using gevent.