HypothesisWorks / hypothesis

Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
https://hypothesis.works
Other
7.61k stars 587 forks source link

some of the tests take longer than expected on slow machines -> FAIL #960

Closed ignatenkobrain closed 7 years ago

ignatenkobrain commented 7 years ago
=================================== FAILURES ===================================
_______________________ test_argon2i_str_and_verify_fail _______________________
    @given(text(alphabet=PASSWD_CHARS, min_size=5, max_size=20),
>          integers(min_value=4,
                    max_value=6),
           integers(min_value=1024 * 1024,
                    max_value=16 * 1024 * 1024)
           )
    @settings(deadline=1500, max_examples=20)
    def test_argon2i_str_and_verify_fail(password, ops, mem):
tests/test_pwhash.py:437: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/site-packages/hypothesis/executors.py:58: in default_new_style_executor
    return function(data)
/usr/lib/python2.7/site-packages/hypothesis/core.py:136: in run
    return test(*args, **kwargs)
tests/test_pwhash.py:437: in test_argon2i_str_and_verify_fail
    integers(min_value=4,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
args = ('   ƒ𣁬', 6, 15695904), kwargs = {}, start = 1509609696.955598
result = None, runtime = 2210.347890853882
    @proxies(test)
    def timed_test(*args, **kwargs):
        self.__test_runtime = None
        start = time.time()
        result = test(*args, **kwargs)
        runtime = (time.time() - start) * 1000
        self.__test_runtime = runtime
        if self.settings.deadline is not_set:
            if (
                not self.__warned_deadline and
                runtime >= 200
            ):
                self.__warned_deadline = True
                note_deprecation((
                    'Test took %.2fms to run. In future the default '
                    'deadline setting will be 200ms, which will '
                    'make this an error. You can set deadline to '
                    'an explicit value of e.g. %d to turn tests '
                    'slower than this into an error, or you can set '
                    'it to None to disable this check entirely.') % (
                        runtime, ceil(runtime / 100) * 100,
                ))
        elif runtime >= self.current_deadline:
>           raise DeadlineExceeded(runtime, self.settings.deadline)
E           DeadlineExceeded: Test took 2210.35ms, which exceeds the deadline of 1500.00ms
/usr/lib/python2.7/site-packages/hypothesis/core.py:624: DeadlineExceeded
---------------------------------- Hypothesis ----------------------------------
Falsifying example: test_argon2i_str_and_verify_fail(password=u'   \u0192\U0002306c', ops=6, mem=15695904)
You can add @seed(302934307671667531413257853548643485645) to this test or run pytest with --hypothesis-seed=302934307671667531413257853548643485645 to reproduce this failure.
__________________________ test_pwhash_str_and_verify __________________________
    @given(text(alphabet=PASSWD_CHARS, min_size=5, max_size=20))
>   @settings(deadline=1500, max_examples=5)
    def test_pwhash_str_and_verify(password):
tests/test_pwhash.py:451: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/site-packages/hypothesis/executors.py:58: in default_new_style_executor
    return function(data)
/usr/lib/python2.7/site-packages/hypothesis/core.py:136: in run
    return test(*args, **kwargs)
tests/test_pwhash.py:451: in test_pwhash_str_and_verify
    @settings(deadline=1500, max_examples=5)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
args = ('     ',), kwargs = {}, start = 1509609754.519142, result = None
runtime = 6857.541084289551
    @proxies(test)
    def timed_test(*args, **kwargs):
        self.__test_runtime = None
        start = time.time()
        result = test(*args, **kwargs)
        runtime = (time.time() - start) * 1000
        self.__test_runtime = runtime
        if self.settings.deadline is not_set:
            if (
                not self.__warned_deadline and
                runtime >= 200
            ):
                self.__warned_deadline = True
                note_deprecation((
                    'Test took %.2fms to run. In future the default '
                    'deadline setting will be 200ms, which will '
                    'make this an error. You can set deadline to '
                    'an explicit value of e.g. %d to turn tests '
                    'slower than this into an error, or you can set '
                    'it to None to disable this check entirely.') % (
                        runtime, ceil(runtime / 100) * 100,
                ))
        elif runtime >= self.current_deadline:
>           raise DeadlineExceeded(runtime, self.settings.deadline)
E           DeadlineExceeded: Test took 6857.54ms, which exceeds the deadline of 1500.00ms
/usr/lib/python2.7/site-packages/hypothesis/core.py:624: DeadlineExceeded
---------------------------------- Hypothesis ----------------------------------
Falsifying example: test_pwhash_str_and_verify(password=u'     ')
You can add @seed(302934307671667531413257853548643485645) to this test or run pytest with --hypothesis-seed=302934307671667531413257853548643485645 to reproduce this failure.
============== 2 failed, 4518 passed, 4 skipped in 452.28 seconds ==============

Well, those machines are slow..

ignatenkobrain commented 7 years ago

whoops, I filed in a wrong place.