holoviz / holoviews

With Holoviews, your data visualizes itself.
https://holoviews.org
BSD 3-Clause "New" or "Revised" License
2.66k stars 396 forks source link

Parallelize unit test #6234

Closed hoxbro closed 1 month ago

hoxbro commented 1 month ago

Fixes #5565

This time it has been in a much better state than when I looked over a year ago. I think this relates to work done in #5954 and #6200, where I made it possible to run tests with required dependencies + pytest.

The main change in this PR is to reset hv.Store before each run.

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 97.91667% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 88.47%. Comparing base (945e179) to head (26f7d46). Report is 5 commits behind head on main.

Files Patch % Lines
holoviews/tests/ipython/test_magics.py 85.71% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #6234 +/- ## ========================================== + Coverage 88.39% 88.47% +0.08% ========================================== Files 323 323 Lines 67620 67632 +12 ========================================== + Hits 59770 59837 +67 + Misses 7850 7795 -55 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

hoxbro commented 1 month ago

Locally, I have run the following script which uses pytest-random-order on https://github.com/holoviz/holoviews/pull/6234/commits/f0541c6cd2cbae2c65166003f1c2b12bdda5d4f9

mkdir -p logs
for i in {01..100}; do
    echo "Run $i"
    pytest holoviews/tests -n logical --dist loadgroup --random-order --color=no > logs/log$i.txt
done

This resulted in the following:

❯ rg FAILED logs | sort
logs/log12.txt:FAILED holoviews/tests/ipython/test_magics.py::TestOptsMagic::test_cell_opts_plot_dynamic
logs/log28.txt:FAILED holoviews/tests/plotting/bokeh/test_callbacks.py::TestCallbacks::test_callback_cleanup
logs/log56.txt:FAILED holoviews/tests/ipython/test_magics.py::TestOptsMagic::test_cell_opts_plot_dynamic
logs/log57.txt:FAILED holoviews/tests/plotting/bokeh/test_callbacks.py::TestCallbacks::test_callback_cleanup
logs/log61.txt:FAILED holoviews/tests/plotting/bokeh/test_callbacks.py::TestCallbacks::test_callback_cleanup
logs/log72.txt:FAILED holoviews/tests/plotting/bokeh/test_server.py::TestBokehServer::test_server_dynamicmap_with_stream
logs/log78.txt:FAILED holoviews/tests/plotting/bokeh/test_server.py::TestBokehServer::test_server_dynamicmap_with_dims
logs/log81.txt:FAILED holoviews/tests/plotting/bokeh/test_server.py::TestBokehServer::test_server_dynamicmap_with_dims
logs/log94.txt:FAILED holoviews/tests/ipython/test_magics.py::TestOptsMagic::test_cell_opts_style_dynamic

❯ rg FAILED logs --json | jq '[select(.type == "match")][].data.path.text' | xargs rg seed | sort
logs/log12.txt:Using --random-order-seed=94455
logs/log28.txt:Using --random-order-seed=484446
logs/log56.txt:Using --random-order-seed=735748
logs/log57.txt:Using --random-order-seed=876950
logs/log61.txt:Using --random-order-seed=72331
logs/log72.txt:Using --random-order-seed=438329
logs/log78.txt:Using --random-order-seed=47376
logs/log81.txt:Using --random-order-seed=747979
logs/log94.txt:Using --random-order-seed=479472

Running with those seeds again, the test did not fail, so I have marked them as flaky.

hoxbro commented 1 month ago

Isolated test failures (from https://github.com/holoviz/holoviews/issues/5565)

Status Command
:green_circle: pytest holoviews/tests -k InspectorTests
:red_circle: * pytest holoviews/tests/core/data/test_imageinterface.py::RGBElement_ImageInterfaceTests::test_reduce_to_single_values
:green_circle: holoviews/tests/core/test_options.py::TestCrossBackendOptionSpecification::test_mpl_bokeh_mpl_via_builders_opts_method

*) Though, it is now consistently failing...