rjdbcm / Aspidites

The reference implementation of the Woma Programming Language compiler.
http://aspidites.org
GNU General Public License v3.0
2 stars 1 forks source link

Flakey monad #13

Closed rjdbcm closed 3 years ago

rjdbcm commented 3 years ago

This flaky call I cannot reproduce outside of hypothesis. I am opening this issue on the off chance someone else can.

=================================== FAILURES ===================================
__________________________ test_integer_monad_sanity ___________________________

    @given(x=st.integers(min_value=MIN, max_value=MAX))
>   def test_integer_monad_sanity(x):

Aspidites/tests/test_aspidites.py:179: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <hypothesis.core.StateForActualGivenExecution object at 0x122499ca0>
message = 'Hypothesis test_integer_monad_sanity(x=0) produces unreliable results: Falsified on the first call but did not on a subsequent one'

    def __flaky(self, message):
        if len(self.falsifying_examples) <= 1:
>           raise Flaky(message)
E           hypothesis.errors.Flaky: Hypothesis test_integer_monad_sanity(x=0) produces unreliable results: Falsified on the first call but did not on a subsequent one

../../../.local/share/virtualenvs/Aspidites-_en4Sn-i/lib/python3.9/site-packages/hypothesis/core.py:890: Flaky
---------------------------------- Hypothesis ----------------------------------
Falsifying example: test_integer_monad_sanity(
    x=0,
)
Unreliable test timings! On an initial run, this test took 252.86ms, which exceeded the deadline of 200.00ms, but on a subsequent run it took 15.11 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.

You can reproduce this example by temporarily adding @reproduce_failure('6.23.2', b'AAAAAAA=') as a decorator on your test case
rjdbcm commented 3 years ago

Latest 25.10+ patches have fixed these issues for now.