search

python-adaptive / adaptive

:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions
http://adaptive.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.17k stars 60 forks source link

learner tests fail #55

Open basnijholt opened 5 years ago

basnijholt commented 5 years ago

(original issue on GitLab)

opened by Joseph Weston (@jbweston) at 2018-07-09T15:41:24.516Z

Currently several integrator learner tests fail.

To reproduce checkout current master (f268c8d as of now)

Running the test suite with the following random seeds will illustrate all of the failure modes:

pytest --randomly-dont-reorganize --randomly-seed=1531149508 adaptive/tests/test_cquad.py  # 2 failures in integrator learner
pytest --randomly-dont-reorganize --randomly-seed=1531150331 adaptive/tests/test_cquad.py  # 1 same failure as above, and 1 different

I will mark all these tests as xfailing for now, but IMO at least 1 of these failures does indicate a bug in the interval logic.

basnijholt commented 5 years ago

originally posted by Joseph Weston (@jbweston) at 2018-07-10T09:40:34.532Z on GitLab

Found a seed where a learner2D test fails (under rescaling) on commit f268c8d:

pytest --randomly-dont-reorganize --randomly-seed=1531149508 adaptive/tests/test_learner.py

@anton-akhmerov points out that because of gitlab:#83 we should just mark this as xfailing now

basnijholt commented 5 years ago

originally posted by Joseph Weston (@jbweston) at 2018-07-10T09:44:42.300Z on GitLab

And another one, also on f268c8d

pytest --randomly-dont-reorganize --randomly-seed=1531215727 adaptive/tests/test_learner.py
basnijholt commented 5 years ago

originally posted by Joseph Weston (@jbweston) at 2018-07-10T09:52:18.983Z on GitLab

@basnijholt check this out!

It only fails on some seeds, and the reason that we didn't see it before was probably because the pipelines were always failing and we didn't check/notice which individual tests were failing

basnijholt commented 5 years ago

originally posted by Bas Nijholt (@basnijholt) at 2018-09-24T14:26:06.653Z on GitLab

test_expected_loss_improvement_is_less_than_total_loss for the Learner2D sometimes fails and sometimes passes. I xfailed it for now in https://gitlab.kwant-project.org/qt/adaptive/commit/eebc5accb225ee570d7419eff57cd351fbd5851c. pytest --randomly-dont-reorganize --randomly-seed=1537797311