Open weiji14 opened 3 years ago
Ok, the flakiness appears to have been an upstream GMT issue that was fixed in https://github.com/GenericMappingTools/gmt/pull/3344. There are some tests that are wrong but currently passing (i.e. false positives) identified in https://github.com/GenericMappingTools/pygmt/issues/1217#issuecomment-827847551 and https://github.com/GenericMappingTools/pygmt/issues/1217#issuecomment-827847551 that need to be updated once we bump to GMT 6.2.0rc2.
The past few flaky tests revealing GMT bugs have convinced me of the usefulness of the current structure, even though it would be nice to have the option to run tests in parallel.
We seem to be semi-regularly getting failures on windows-latest - Python 3.7 / NumPy 1.18 with : ..\tests\test_sph2grd.py::test_sph2grd_outgrid FAILED [ 87%] ..\tests\test_sph2grd.py::test_sph2grd_no_outgrid FAILED [ 87%] due to issues with the remote file.
We seem to be semi-regularly getting failures on windows-latest - Python 3.7 / NumPy 1.18 with : ..\tests\test_sph2grd.py::test_sph2grd_outgrid FAILED [ 87%] ..\tests\test_sph2grd.py::test_sph2grd_no_outgrid FAILED [ 87%] due to issues with the remote file.
Yes this has been popping up recently, but I don't think this is related to flakiness in a single GMT session since the error is Error: [ERROR]: Libcurl Error: Timeout was reached
, so maybe open a separate issue for this.
https://forum.generic-mapping-tools.org/t/memory-temporary-storage-issues/5256 This post is a good example showing that using a single GMT session sometimes causes issues.
Description of the problem
There's been instances of flaky tests in PyGMT's test suite reported in https://github.com/GenericMappingTools/pygmt/issues/1217#issuecomment-825561365. This likely stems from the fact that PyGMT uses a single GMT session (initiated during
import pygmt
) instead of separate GMT sessions for each figure (see https://github.com/GenericMappingTools/pygmt/pull/327#issuecomment-541782890).@meghanrjones asked about whether we should stick with using a single GMT session in https://github.com/GenericMappingTools/pygmt/issues/1217#issuecomment-827230998, or use independent sessions per figure
Full code that generated the error
Flaky tests are hard to reproduce (that is their definition actually), but in PyGMT's case, can be found e.g. when a single test passing on
pytest pygmt/tests/test_somemodule.py
fails when ran usingmake test
, or vice versa.E.g. as reported by @meghanrjones in https://github.com/GenericMappingTools/pygmt/issues/1217#issuecomment-825753479
Related issues affected by having a single GMT session:
217
733
1582
System information
Please paste the output of
python -c "import pygmt; pygmt.show_versions()"
: