search

initc3 / HoneyBadgerMPC

Robust MPC-based confidentiality layer for blockchains
GNU General Public License v3.0
128 stars 64 forks source link

Non-deterministic test failures #415

Open sbellem opened 4 years ago

sbellem commented 4 years ago

Some tests appear to fail in non-deterministic ways, meaning that for the exact same code version/hash a test sometimes passes and sometimes fails.

I'll soon provide examples, and will look into how we may possibly address the problem.

Resources

The trio library may be worth looking at, at least conceptually, for how it handles such non-deterministic failures. From their contributing section docs:

Tests MUST pass deterministically. Flakey tests make for miserable developers. One common source of indeterminism is scheduler ordering; if you’re having trouble with this, then trio.testing provides powerful tools to help control ordering, like trio.testing.wait_all_tasks_blocked(), trio.testing.Sequencer, and trio.testing.MockClock (usually used as a fixture: async def test_whatever(autojump_clock): ...). And if you need more tools than this then we should add them.

In addition to the interesting test tools to help prevent non-deterministic tests failures, it is in interesting to see how they handle these test failures when they do occur, on Travis CI for instance:

Another interesting issue is on finding and reproducing heisenbugs in concurrent programs, it has many links to interesting resource:

sbellem commented 4 years ago

Example:

docker run -it $tag \
    pytest --verbose \
           --numprocesses=auto \
           --cov \
           --cov-report=term-missing \
           --cov-report=xml \
           -Wignore::DeprecationWarning

/opt/venv/lib/python3.7/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.

  warner(PytestBenchmarkWarning(text))

============================= test session starts ==============================

platform linux -- Python 3.7.5, pytest-5.3.1, py-1.8.0, pluggy-0.13.1 -- /opt/venv/bin/python
cachedir: .pytest_cache
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/usr/src/HoneyBadgerMPC/.hypothesis/examples')
rootdir: /usr/src/HoneyBadgerMPC, inifile: pytest.ini
plugins: mock-1.13.0, web3-5.4.0, asyncio-0.10.0, xdist-1.30.0, benchmark-3.2.2, cov-2.8.1, forked-1.1.3, env-0.6.2, hypothesis-4.50.6
[gw0] linux Python 3.7.5 cwd: /usr/src/HoneyBadgerMPC
[gw1] linux Python 3.7.5 cwd: /usr/src/HoneyBadgerMPC
[gw0] Python 3.7.5 (default, Nov 23 2019, 06:10:46)  -- [GCC 8.3.0]
[gw1] Python 3.7.5 (default, Nov 23 2019, 06:10:46)  -- [GCC 8.3.0]
gw0 [260] / gw1 [260]
scheduling tests via LoadScheduling
tests/test_asynchromix.py::test_phase1 
tests/test_asynchromix.py::test_butterfly_network Fatal Python error: Illegal instruction
Thread 0x00007f0e63472700 (most recent call first):
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 400 in read
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 432 in from_io
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn

Current thread 0x00007f0e6395f740 (most recent call first):

  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 224 in _write_polys
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 357 in generate_values
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 606 in _generate
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 633 in generate_powers
  File "/usr/src/HoneyBadgerMPC/tests/test_asynchromix.py", line 48 in test_phase1
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88 in _run
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 1763 in _run_once
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 534 in run_forever
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 566 in run_until_complete
  File "/opt/venv/lib/python3.7/site-packages/pytest_asyncio/plugin.py", line 158 in pytest_pyfunc_call
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/_pytest/python.py", line 1435 in runtest
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 131 in pytest_runtest_call
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 207 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 234 in from_call
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 207 in call_runtest_hook
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 182 in call_and_report
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 96 in runtestprotocol
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 81 in pytest_runtest_protocol
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/xdist/remote.py", line 87 in run_one_test
  File "/opt/venv/lib/python3.7/site-packages/xdist/remote.py", line 70 in pytest_runtestloop
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/_pytest/main.py", line 246 in _main
  File "/opt/venv/lib/python3.7/site-packages/_pytest/main.py", line 196 in wrap_session
  File "/opt/venv/lib/python3.7/site-packages/_pytest/main.py", line 239 in pytest_cmdline_main
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/xdist/remote.py", line 258 in <module>
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 1084 in executetask
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 1060 in serve
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 1554 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>
[gw1] node down: Not properly terminated
[gw1] [  0%] FAILED tests/test_asynchromix.py::test_phase1 

replacing crashed worker gw1