dask / dask

Parallel computing with task scheduling
https://dask.org
BSD 3-Clause "New" or "Revised" License
12.3k stars 1.69k forks source link

Can't find `cleanup` pytest fixture #9137

Open jrbourbeau opened 2 years ago

jrbourbeau commented 2 years ago

We started seeing

___________ ERROR at setup of test_blockwise_different_optimization ____________
[gw2] linux -- Python 3.9.12 /usr/share/miniconda3/envs/test-environment/bin/python
file /home/runner/work/dask/dask/dask/tests/test_distributed.py, line 472
  def test_blockwise_different_optimization(c):
file /usr/share/miniconda3/envs/test-environment/lib/python3.9/site-packages/distributed/utils_test.py, line 617
  @pytest.fixture
  def client(loop, cluster_fixture):
file /usr/share/miniconda3/envs/test-environment/lib/python3.9/site-packages/distributed/utils_test.py, line 138
  @pytest.fixture
  def loop(cleanup):
E       fixture 'cleanup' not found
>       available fixtures: c, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cluster_fixture, cov, doctest_namespace, loop, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, shuffle_method, testrun_uid, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id
>       use 'pytest --fixtures [testpath]' for help on them.

/usr/share/miniconda3/envs/test-environment/lib/python3.9/site-packages/distributed/utils_test.py:138

on main in all our distributed-related tests. Probably related to https://github.com/dask/distributed/pull/6231 cc @graingert

jrbourbeau commented 2 years ago

Looks like adding the cleanup fixture to the loop fixture

https://github.com/dask/distributed/blob/5feb17151cdf660a3443abf8596444a9f51dc575/distributed/utils_test.py#L138-L139

is what triggered this. Though not immediately sure why that's a problem

keewis commented 2 years ago

Though not immediately sure why that's a problem

maybe changing the order of definition helps? cleanup is defined after loop at the moment (which should not be an issue with standard python but I'm not sure what exactly the fixture decorator does).

graingert commented 2 years ago

Looks like adding the cleanup fixture to the loop fixture

https://github.com/dask/distributed/blob/5feb17151cdf660a3443abf8596444a9f51dc575/distributed/utils_test.py#L138-L139

is what triggered this. Though not immediately sure why that's a problem

Fixtures are discovered by pytest by looking at specially marked objects returned by the fixture decorator in the modules, running the fixture decorator itself doesn't register fixtures.

If you get pytest to discover a fixture by importing it, you also need to import all its dependencies so pytest can discover them too