Closed oskooi closed 1 year ago
How does assertClose
work for arrays? One of the major reasons we are using compare_arrays
is because the Python array-comparison methods are element-wise rather than using array norms, which made it impossible to reliably specify a relative error.
(Basically, I think Python's isclose
and allclose
are broken.)
How does assertClose work for arrays? One of the major reasons we are using compare_arrays is because the Python array-comparison methods are element-wise rather than using array norms, which made it impossible to reliably specify a relative error.
assertClose
is the custom array-comparator function which uses the array norm to define the relative error that we added as part of #1569:
I suspect the current use of compare_arrays
and numpy.testing.assert_allclose
in the MPB unit tests are why this test is so flaky (see recent failure). This PR seems to improve things.
Inspecting the log of the failing test_mpb.py
instance, the cause of the error is numpy.testing.assert_allclose
(and not compare_arrays
):
======================================================================
FAIL: test_get_eigenvectors (__main__.TestModeSolver)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/meep/meep/build/meep-1.28.0-beta/_build/sub/python/../../../python/tests/test_mpb.py", line 1609, in test_get_eigenvectors
compare_eigenvectors("tutorial-te-eigenvectors.h5", 1, 8)
File "/home/runner/work/meep/meep/build/meep-1.28.0-beta/_build/sub/python/../../../python/tests/test_mpb.py", line 1606, in compare_eigenvectors
np.testing.assert_allclose(expected, ev, rtol=1e-3)
File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 1592, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 862, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=0
This is perhaps not surprising since numpy.testing.assert_allclose
tries to enforce equivalence among all elements of the two arrays and fails if any one element disagrees. At a minimum, we can therefore simply replace the one instance of this function in test_mpb.py
with assertClose
.
However, it is not clear why we need to have both compare_arrays
and assertClose
since they perform the same function using arrays norms yet in a slightly different way.
The MPB unit tests in
test_mpb.py
are currently usingnumpy.testing.assert_allclose
andcompare_arrays
(added in #345). For comparing two arrays, these functions have been replaced withassertClose
which is used in all the other tests.This PR updates
test_mpb.py
to only useassertClose
. This modification should hopefully make this test less flaky (it is currently failing the CI). Also,compare_arrays
is removed fromutils.py
since it is not being used anywhere else. Finally, type hints are added toassertClose
and its docstrings are updated.