Tests cleaning - Githubissues

obackhouse commented 2 years ago

Changes:

Replaces the old system cache with the new one
Adds pytest.mark decorators for fast, slow and veryslow tests (equivalent to test levels 0, 2 and 3, where level 1 is undecorated)
run_test.py now uses pytest to dispatch the tests which is more robust and has a nicer output.

Some things to be addressed before merging this into master:

[x] @cjcscott: test failures in a few files in edmet/:

ERROR: test_h2o_ccpvdz_EBCCSD_IAO_2occ (__main__.MolecularEDMETTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "vayesta/tests/edmet/test_edmet_molecule.py", line 142, in test_h2o_ccpvdz_EBCCSD_IAO_2occ
emb.kernel()
File "/home/olli/git/vayesta/vayesta/edmet/edmet.py", line 118, in kernel
err = electron_err(cpt)
File "/home/olli/git/vayesta/vayesta/edmet/edmet.py", line 115, in electron_err
err = self.calc_electron_number_defect(cpt, nelec_mf, sym_parents, nsym)
File "/home/olli/git/vayesta/vayesta/edmet/edmet.py", line 281, in calc_electron_number_defect
result = frag.kernel(construct_bath=construct_bath, chempot=chempot)
File "/home/olli/git/vayesta/vayesta/edmet/fragment.py", line 684, in kernel
cluster_solver.kernel(eris=eris)
File "/home/olli/git/vayesta/vayesta/solver/ebccsd.py", line 65, in kernel
inp, couplings = self.get_input(eris)
File "/home/olli/git/vayesta/vayesta/solver/ebccsd.py", line 42, in get_input
if self.opts.v_ext is not None:
AttributeError: 'Options' object has no attribute 'v_ext'

[x] @cjcscott: can you address #9 and re-enable the relevant commented tests?

[x] @Majokux: test failures in ewf/test_h2_pbc:


FAIL: test_dm1 (__main__.Test_UMP2)
----------------------------------------------------------------------
Traceback (most recent call last):
File "vayesta/tests/ewf/test_h2_pbc.py", line 79, in test_dm1
self.assertAllclose(dm1, dm1_exact)
File "/home/olli/git/vayesta/vayesta/tests/common.py", line 26, in assertAllclose
self.assertAllclose(actual[i], desired[i], rtol=rtol, atol=atol, **kwargs)
File "/home/olli/git/vayesta/vayesta/tests/common.py", line 29, in assertAllclose
np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, **kwargs)
File "/home/olli/.local/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 1529, in assert_allclose
verbose=verbose, header=header, equal_nan=equal_nan)
File "/home/olli/.local/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 842, in assert_array_compare
raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-08, atol=1e-07

Mismatched elements: 2 / 81 (2.47%) Max absolute difference: 1.31129773e-07 Max relative difference: 6.04587939e-07 x: array([[ 0.060869, 0.143332, 0.057767, -0.00951 , -0.020941, -0.009388, -0.00951 , -0.020941, -0.009388], [ 0.143332, 0.369976, 0.143332, -0.020941, -0.045602, -0.020941, -0.020941, -0.045602, -0.020941], [ 0.057767, 0.143332, 0.060869, -0.009388, -0.020941, -0.00951 , -0.009388, -0.020941, -0.00951 ],... y: array([[ 0.060869, 0.143332, 0.057767, -0.00951 , -0.020941, -0.009388, -0.00951 , -0.020941, -0.009388], [ 0.143332, 0.369976, 0.143332, -0.020941, -0.045602, -0.020941, -0.020941, -0.045602, -0.020941], [ 0.057767, 0.143332, 0.060869, -0.009388, -0.020941, -0.00951 , -0.009388, -0.020941, -0.00951 ],...

- [x] @Majokux: test failures in `ewf/test_solid`:

FAIL: test_restricted (main.SolidEWFTests) Tests restriced EWF for a solid, binary system with and without k-point folding.

Traceback (most recent call last): File "vayesta/tests/ewf/test_solid.py", line 39, in test_restricted self.assertAllclose(kemb.e_tot, e_expected) File "/home/olli/git/vayesta/vayesta/tests/common.py", line 29, in assertAllclose np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, **kwargs) File "/home/olli/.local/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 1529, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/home/olli/.local/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 842, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-08, atol=1e-07

Mismatched elements: 1 / 1 (100%) Max absolute difference: 0.06151238 Max relative difference: 0.00762305 x: array(-8.130774) y: array(-8.069262)


- [ ] Check tests running on all platforms? I guess I can also enable the CI now since this will be public soon.
- [ ] Test formatting? I don't think this is super necessary for now, but some of the test files have got a bit messy. Maybe another time.

maxnus commented 2 years ago

Just to check, I also get

======================================================================
FAIL: test_dmet_energy (__main__.Test_UCCSD)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_h2_pbc.py", line 127, in test_dmet_energy
    self.assertAllclose(etot_dmet, e_ref, rtol=0)
  File "/home/max/git/vayesta/vayesta/tests/common.py", line 29, in assertAllclose
    np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/max/.local/lib/python3.8/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0, atol=1e-07

Mismatched elements: 1 / 1 (100%)
Max absolute difference: 0.68055945
Max relative difference: 0.39284262
 x: array(-2.412957)
 y: array(-1.732397)

in ewt/test_h2_pbc.py which wasn't listed above - did you get this too?

maxnus commented 2 years ago

Just to check, I also get

======================================================================
FAIL: test_dmet_energy (__main__.Test_UCCSD)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_h2_pbc.py", line 127, in test_dmet_energy
    self.assertAllclose(etot_dmet, e_ref, rtol=0)
  File "/home/max/git/vayesta/vayesta/tests/common.py", line 29, in assertAllclose
    np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/max/.local/lib/python3.8/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0, atol=1e-07

Mismatched elements: 1 / 1 (100%)
Max absolute difference: 0.68055945
Max relative difference: 0.39284262
 x: array(-2.412957)
 y: array(-1.732397)

in ewt/test_h2_pbc.py which wasn't listed above - did you get this too?

OK that one should be fixed now ... which wasn't even on the original list

maxnus commented 2 years ago

Test_UMP2.test_dm1 in ewf/test_h2_pbc.py should now pass - I tightened the convergence tolerance of the mean-field a bit

obackhouse commented 2 years ago

Just to check, I also get

======================================================================
FAIL: test_dmet_energy (__main__.Test_UCCSD)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_h2_pbc.py", line 127, in test_dmet_energy
    self.assertAllclose(etot_dmet, e_ref, rtol=0)
  File "/home/max/git/vayesta/vayesta/tests/common.py", line 29, in assertAllclose
    np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/max/.local/lib/python3.8/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0, atol=1e-07

Mismatched elements: 1 / 1 (100%)
Max absolute difference: 0.68055945
Max relative difference: 0.39284262
 x: array(-2.412957)
 y: array(-1.732397)

in ewt/test_h2_pbc.py which wasn't listed above - did you get this too?

Yes sorry, forgot to include that

obackhouse commented 2 years ago

@cjcscott does https://github.com/BoothGroup/Vayesta/pull/17/commits/81ae52fd1018940d302e92063dc7060ddf62aebc reduce coverage? Want me to redo that test for a converging system?

maxnus commented 2 years ago

I fixed the test failures in ewf/test_solid by deleting ewf/test_solid *taps head*

maxnus commented 2 years ago

I will merge this now, we don't want to continue developing in this PR - most of these task were unrelated to the pytest migration and new testsytems anyway. If there are open task, we can open separate issues or PRs

BoothGroup / Vayesta

Tests cleaning #17

FAIL: test_restricted (main.SolidEWFTests) Tests restriced EWF for a solid, binary system with and without k-point folding.