search

scikit-learn-contrib / scikit-matter

A collection of scikit-learn compatible utilities that implement methods born out of the materials science and chemistry communities
https://scikit-matter.readthedocs.io/en/v0.2.0/
BSD 3-Clause "New" or "Revised" License
76 stars 20 forks source link

Some tests seems to run multiple times #134

Closed Luthaf closed 1 year ago

Luthaf commented 2 years ago

From https://github.com/lab-cosmo/scikit-cosmo/runs/6552874752?check_suite_focus=true

GLOB sdist-make: /home/runner/work/scikit-cosmo/scikit-cosmo/setup.py
tests create: /home/runner/work/scikit-cosmo/scikit-cosmo/.tox/tests
tests installdeps: coverage[toml], parameterized
tests inst: /home/runner/work/scikit-cosmo/scikit-cosmo/.tox/.tmp/package/1/skcosmo-0.1.1.zip
tests installed: coverage==6.4,joblib==1.1.0,numpy==1.22.4,parameterized==0.8.1,scikit-learn==1.1.1,scipy==1.8.1,skcosmo @ file:///home/runner/work/scikit-cosmo/scikit-cosmo/.tox/.tmp/package/1/skcosmo-0.1.1.zip,threadpoolctl==3.1.0,tomli==2.0.1
tests run-test-pre: PYTHONHASHSEED='920532320'
tests run-test: commands[0] | coverage run -m unittest discover -p '*.py'
....................................................../home/runner/work/scikit-cosmo/scikit-cosmo/.tox/tests/lib/python3.9/site-packages/sklearn/linear_model/_ridge.py:251: UserWarning: Singular matrix in solving dual problem. Using least-squares solution instead.
  warnings.warn(
.............................................................................../home/runner/work/scikit-cosmo/scikit-cosmo/.tox/tests/lib/python3.9/site-packages/skcosmo/utils/_orthogonalizers.py:57: UserWarning: Column vector contains only zeros.
  warnings.warn("Column vector contains only zeros.")
.........../home/runner/work/scikit-cosmo/scikit-cosmo/.tox/tests/lib/python3.9/site-packages/skcosmo/_selection.py:214: UserWarning: Score threshold of 0.05 reached.Terminating search at 7 / 10.
  warnings.warn(
/home/runner/work/scikit-cosmo/scikit-cosmo/.tox/tests/lib/python3.9/site-packages/skcosmo/_selection.py:214: UserWarning: Score threshold of 0.4 reached.Terminating search at 6 / 10.
  warnings.warn(
............................/home/runner/work/scikit-cosmo/scikit-cosmo/.tox/tests/lib/python3.9/site-packages/skcosmo/_selection.py:214: UserWarning: Score threshold of 0.05 reached.Terminating search at 7 / 10.
  warnings.warn(
/home/runner/work/scikit-cosmo/scikit-cosmo/.tox/tests/lib/python3.9/site-packages/skcosmo/_selection.py:214: UserWarning: Score threshold of 0.4 reached.Terminating search at 6 / 10.
  warnings.warn(
........./home/runner/work/scikit-cosmo/scikit-cosmo/.tox/tests/lib/python3.9/site-packages/skcosmo/_selection.py:214: UserWarning: Score threshold of 0.05 reached.Terminating search at 7 / 10.
  warnings.warn(
/home/runner/work/scikit-cosmo/scikit-cosmo/.tox/tests/lib/python3.9/site-packages/skcosmo/_selection.py:214: UserWarning: Score threshold of 0.4 reached.Terminating search at 6 / 10.
  warnings.warn(
.
----------------------------------------------------------------------
Ran 182 tests in 69.540s

OK

The UserWarning: Score threshold of 0.4 reached.Terminating search at 6 / 10. should only appear once, but is there three times.

Changing https://github.com/lab-cosmo/scikit-cosmo/blob/f433e28a8e5ff13f2dd1aa9a9f2dd2fc3f606218/pyproject.toml#L19 to coverage run -m unittest discover -p "test_sample_simple_fps.py" makes the test run only once, so there might be some strange interaction between unittest, discover and coverage.

Luthaf commented 1 year ago

From the latest CI run with pytest:


tests/test_sample_simple_fps.py::TestFPS::test_threshold
tests/test_voronoi_fps.py::TestFPS::test_threshold
tests/test_voronoi_fps.py::TestVoronoiFPS::test_threshold
  /home/runner/work/scikit-matter/scikit-matter/src/skmatter/_selection.py:211: UserWarning: Score threshold of 0.4 reached.Terminating search at 6 / 10.
    warnings.warn(

So it seems like the same warning was sent by multiple tests, throwing me off initially.