Test on scipy 1.7

[jarrodmillman]

SciPy 1.7 had a lot of updates to scipy.stats and it looks like it broke our tests. You can see the release notes here: https://scipy.github.io/devdocs/release.1.7.0.html

I pinned to scipy>=1.6,<1.7 in #198. So the main branch passes the CI tests. You can make PRs off main (and update your existing PRs). But we should drop support for scipy 1.6 soon and figure out whether we need to update our tests or whether we are uncovering bugs in scipy.stats, which were introduced in 1.7.

[jarrodmillman]

Our tests pass on scipy 1.6.

On scipy 1.7.3, we get

    def test_binom_conf_interval():
        res = binom_conf_interval(10, 3)
        expected = (0.05154625578928545, 0.6915018049393984)
>       np.testing.assert_equal(res, expected)
E       AssertionError: 
E       Items are not equal:
E       item=0
E       
E        ACTUAL: 0.05154625578928546
E        DESIRED: 0.05154625578928545

permute/tests/test_utils.py:24: AssertionError

On scipy 1.8.1, we get

___________________________ test_binom_conf_interval ___________________________

    def test_binom_conf_interval():
        res = binom_conf_interval(10, 3)
        expected = (0.05154625578928545, 0.6915018049393984)
>       np.testing.assert_equal(res, expected)
E       AssertionError: 
E       Items are not equal:
E       item=0
E       
E        ACTUAL: 0.05154625578928546
E        DESIRED: 0.05154625578928545

permute/tests/test_utils.py:24: AssertionError
_________________________ test_hypergeom_conf_interval _________________________

    def test_hypergeom_conf_interval():
        res = hypergeom_conf_interval(2, 1, 5, cl=0.95, alternative="two-sided")
        expected = (1.0, 4.0)
        np.testing.assert_equal(res, expected)

        res2 = hypergeom_conf_interval(2, 1, 5, cl=0.95, alternative="upper")
        expected2 = (0.0, 4.0)
        np.testing.assert_equal(res2, expected2)

        res3 = hypergeom_conf_interval(2, 1, 5, cl=0.95, alternative="lower")
        expected3 = (1.0, 5.0)
        np.testing.assert_equal(res3, expected3)

        res4 = hypergeom_conf_interval(2, 2, 5, cl=0.95, alternative="two-sided")
        expected4 = (2.0, 5.0)
        np.testing.assert_equal(res4, expected4)

        cl = 0.95
        n = 10
        x = 5
        N = 20
        [lot, hit] = [6, 14]
        alternative = "two-sided"
        [lo, hi] = hypergeom_conf_interval(n, x, N, cl=cl, alternative=alternative)
        np.testing.assert_equal(lo, lot)
>       np.testing.assert_equal(hi, hit)
E       AssertionError: 
E       Items are not equal:
E        ACTUAL: 15
E        DESIRED: 14

permute/tests/test_utils.py:68: AssertionError

On scipy 1.9.1, we get:

___________________________ test_binom_conf_interval ___________________________

    def test_binom_conf_interval():
        res = binom_conf_interval(10, 3)
        expected = (0.05154625578928545, 0.6915018049393984)
>       np.testing.assert_equal(res, expected)
E       AssertionError: 
E       Items are not equal:
E       item=0
E       
E        ACTUAL: 0.05154625578928546
E        DESIRED: 0.05154625578928545

permute/tests/test_utils.py:24: AssertionError
_________________________ test_hypergeom_conf_interval _________________________

    def test_hypergeom_conf_interval():
        res = hypergeom_conf_interval(2, 1, 5, cl=0.95, alternative="two-sided")
        expected = (1.0, 4.0)
        np.testing.assert_equal(res, expected)

        res2 = hypergeom_conf_interval(2, 1, 5, cl=0.95, alternative="upper")
        expected2 = (0.0, 4.0)
        np.testing.assert_equal(res2, expected2)

        res3 = hypergeom_conf_interval(2, 1, 5, cl=0.95, alternative="lower")
        expected3 = (1.0, 5.0)
        np.testing.assert_equal(res3, expected3)

        res4 = hypergeom_conf_interval(2, 2, 5, cl=0.95, alternative="two-sided")
        expected4 = (2.0, 5.0)
>       np.testing.assert_equal(res4, expected4)
E       AssertionError: 
E       Items are not equal:
E       item=0
E       
E        ACTUAL: 1
E        DESIRED: 2.0

permute/tests/test_utils.py:58: AssertionError

@mdhaber @tupui Do either of you have any insight on this?

[tupui]

It seems your tests where quite sensitive no? Can you not just relax the tolerances? For the other failures, maybe it's also because it was not a clear cut.