Open drew-parsons opened 3 years ago
@drew-parsons This is with pytest in unstable (for autopackagetest)? I'm not sure, it should work, and the invocation looks fine. I'll look into this though. There were no changes in terms of the actual code (i.e. what is needed for h5py and others), it was mainly getting both pytest <6 and pytest >= 6 to work with the texts.
FYI, I'm happy to accept a PR which adds autopackagetest to the things that need to pass to merge.
Wait, sorry, --with-mpi
shouldn't be given to the pytest which sets up the environment (which uses pytester to call pytest again with and without --with-mpi
). The tree of processes looks like:
pytest -p pytester
-> mpirun -n 2
---> pytest --with-mpi <generated_test_file.py>
Some of the inner pytest tests should fail or error, as that's testing the behaviour of the options when the environment is not set up correctly when running the tests for a project which uses the plugin (e.g. h5py).
I'm also slightly confused by the Xvfb references in the tests, as we don't need it (and it wouldn't surprise me if that plugin was trying to spawn subprocesses, which could really confuse MPI).
The behaviour I'm reporting makes more sense if I shouldn't be providing --with-mpi
to pytest anyway. Tests expected to fail would get acknowledged as xfail.
I agree, the Xvfb errors are rather odd. I'm not sure how to interpret them. What is tox doing differently to pytest launched directly on the command line?
Adding autopkgtest to your test cases is a good idea. I'll push a PR if I can figure out how github manages its CI to handle autopkgtest. It would mean you'd have to have your test machine generate and install .deb packages, or at least have a debian/tests directory available to from. I've switched the debian package over to tox so debci should be less volatile now.
I think it might be constructive to focus attention on the Xvfb error here. I think it's the root cause (or symptom) of the problems. I'm having trouble also running h5py tests --with-mpi, with random successes and failures (running as mpirun -n 4 python3 -c "import h5py; h5py.run_tests('-v --with-mpi -k mpi')"
)
The problem with h5py exhibits as a hang in h5py's tests/test_file.py::TestMPI::test_mpio
. I've noticed that when it hangs, it also emits the Xvfb error, same as we saw above in pytest-mpi itself. When h5py test_mpio does not hang, there is no Xvfb error.
Running the h5py test repeatedly, it seems to suffer the Xvfb error about 50% of the time. My guess is that the randomness might be in the order in which the pytest modules are loaded, whether pytest loads pytest-xvfb or pytest-mpi first.
I can work around the problem by running tests with --no-xvfb
. But is there something that can be done in pytest-mpi to make it more robust when pytest-xvfb is present? (or equivalently, can a patch be identified for pytest-xvfb?)
Incidentally, all the pytest plugins seem get loaded by pytest simply by being present, regardless of whether the specific test at hand uses them. The plugins tag for h5py is
plugins: cov-2.10.1, mpi-0+unknown, doctestplus-0.9.0, remotedata-0.3.2, asyncio-0.14.0, filter-subpackage-0.1.1, arraydiff-0.3, astropy-header-0.1.2, hypothesis-5.43.3, openfiles-0.5.0, xvfb-1.2.0
more or less the same as for pytest-mpi itself. If that's the case then you might be able to reproduce the error (50% of the time) simply by installing pytest-xvfb, if it's not already installed.
Indeed, I can reproduce the Xvfb error reliably just by running mpirun -n 4 pytest-3
in an empty directory (no tests). All plugins are still listed by pytest, including both mpi and xvfb, even without adding --with-mpi
.
@drew-parsons Did you try running with --runpytest=subprocess
? I’m running with pytest tests -p pytester --runpytest=subprocess
and it works when using an installed pytest-mpi
.
However I don’t understand how inplace (i.e. without installing to the system) testing is supposed to work. It fails with __main__.py: error: unrecognized arguments: --with-mpi
, which is somewhat expected if pytest
does not know about the being tested pytest-mpi
. Any idea?
No, it's still giving the same Xvfb error if I run with --runpytest=subprocess
(on a Debian system, pytest-3 tests -p pytester --runpytest=subprocess
)
Ah sorry, I misunderstood your issue. So it’s not really pytest-mpi fault in your case anyway… Curious though, why is there xvfb
in your building environment since it’s not required (both here and for h5py)?
My system has python3-pytest-xvfb. The pytest environment seems to be loading all modules that happen to be available on the system.
The last part should be a pytest issue instead I think. Regarding the first part, I would advise building in chroot/container, it avoids “contamination” by things available on the system. ;)
Well, sure, the official build is in a chroot :)
There seems to be fragility in the pytest-mpi 0.5 tests. I'm building and running on Debian unstable. Possibly I'm not triggering the tests correctly, so let me know if that my problem. These errors occur with
python3 -m pytest -v -p pytester
. I can confirm that tests do pass when run via tox (apart from a manifest error due to the debian subdir).I gather the
--with-mpi
is mandatory, andpython3 -m pytest -v -p pytester
is expected to fail. But I get failure withpython3 -m pytest -v -p pytester --with-mpi
, and a different failure each time depending on the location of the verbose-v
flag.I've copied the tests dir to a separate dir to ensure the installed pytest-mpi is being invoked, not the source code.
Summary, running inside the test dir:
without
-v
,(fails: test_mpi_tmpdir, test_mpi_tmp_path, test_mpi_with_mpi, test_mpi_skip_under_mpi, test_mpi_xfail_under_mpi)
python3 -m pytest -v -p pytester --with-mpi
,python3 -m pytest -p pytester -v --with-mpi
,(a different 4 tests failing)
python3 -m pytest -p pytester --with-mpi -v
,(now only 2 tests failing)
I get different results if I run from the parent dir above the
tests
subdir. Invoking-k test_markers
, all test_markers tests passed, But the success is not reliable. Runningpython3 -m pytest -p pytester --with-mpi -k 'test_markers'
a second time from the parent dir, test_mpi_only_mpi fails. I guess the test cache files ( .pytest_cache) must involved in the failure. But removing .pytest_cache is not sufficient. Repeatingrm -rf .pytest_cache/; python3 -m pytest -p pytester --with-mpi -k 'test_markers'
multiple times from the parent directory, I variously get 4, 3 or 1 error.The failure is apparently random, which suggests it might be related to file flushing, or synchronicity issues. The tests fail rather than hanging.
A sample full error output is
That last message "To run the MPI tests, you need to use subprocesses". Does it mean
python3 -m pytest -p pytester
or`python3 -m pytest -p pytester --with-mpi
is no longer expected to work? Again, I reiterate that the test pass when run viatox
.