search

HypothesisWorks / hypothesis

Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
https://hypothesis.works
Other
7.39k stars 578 forks source link

Failure within `extract_lambda_source()` AKA, your sins are not forgiven! ;) #104

Closed ckaran closed 8 years ago

ckaran commented 8 years ago

I'm running into a problem with hypothesis, which I've copied at the bottom of this issue. I've created a relatively small py.test file that you can use to test the issue on your system. Similar code used to work for me under earlier versions of hypothesis, but something must have changed...

Anyways, here is the test code, and at the bottom is the output I'm getting

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import pytest
from hypothesis import assume
from hypothesis import example
from hypothesis import given
from hypothesis import Settings
from hypothesis import Verbosity
from hypothesis.strategies import basic
from hypothesis.strategies import binary
from hypothesis.strategies import booleans
from hypothesis.strategies import builds
from hypothesis.strategies import complex_numbers
from hypothesis.strategies import decimals
from hypothesis.strategies import dictionaries
from hypothesis.strategies import fixed_dictionaries
from hypothesis.strategies import floats
from hypothesis.strategies import fractions
from hypothesis.strategies import frozensets
from hypothesis.strategies import integers
from hypothesis.strategies import just
from hypothesis.strategies import lists
from hypothesis.strategies import one_of
from hypothesis.strategies import sampled_from
from hypothesis.strategies import sets
from hypothesis.strategies import streaming
from hypothesis.strategies import text
from hypothesis.strategies import tuples

__docformat__ = "restructuredtext en"

##############################################################################
##############################################################################
# hypothesis.strategies that are globally useful
##############################################################################
##############################################################################

_ID = integers(min_value=0, max_value=(2**64 - 1))
_CREATION_DATE = floats(min_value=0.0)
_SIZE = integers(min_value=0, max_value=2**64)
_MSG_GEN = tuples(_ID, _ID, _CREATION_DATE, _SIZE)
_MSG_ACTION = one_of(just(None), _MSG_GEN)
_BANDWIDTH_ACTION = one_of(booleans(), floats(min_value=1.0))
_NUM = integers(min_value=0, max_value=10)
_ARGS = _NUM.flatmap(lambda n:
                     tuples(lists(_MSG_ACTION, min_size=n, max_size=n),
                            lists(_BANDWIDTH_ACTION, min_size=n, max_size=n)))

##############################################################################
##############################################################################
# Minimal test that fails
##############################################################################
##############################################################################

class Test_Hypothesis(object):
    @pytest.mark.slow
    @given(_ARGS, settings=Settings(timeout=1000, min_satisfying_examples=1000))
    def test_one(self, args):
        pass
    # End of test_one()

    @given(_ARGS, settings=Settings(timeout=1000, min_satisfying_examples=1000))
    def test_two(self, args):
        pass
    # End of test_two()

    @given(_ARGS)
    def test_three(self, args):
        pass
    # End of test_three()
# End of class Test_Hypothesis

##############################################################################
##############################################################################
# Main
##############################################################################
##############################################################################

if __name__ == "__main__":
    pass

Output:

Cems-Mac-Pro:Desktop crushed$ py.test -vvvvvvv -s test_Hypothesis.py 
============================================================ test session starts ============================================================
platform darwin -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /opt/local/Library/Frameworks/Python.framework/Versions/3.4/bin/python3.4
rootdir: /Users/crushed/Desktop, inifile: 
plugins: xdist
collected 3 items 

test_Hypothesis.py::Test_Hypothesis::test_one <- .hypothesis/eval_source/hypothesis_temporary_module_2de32ff1d5e50f88f5b700adc3c8602d72e8a202.py FAILED
test_Hypothesis.py::Test_Hypothesis::test_three <- .hypothesis/eval_source/hypothesis_temporary_module_480c901976580cb7f70c301bbce68eeb103f01ba.py FAILED
test_Hypothesis.py::Test_Hypothesis::test_two <- .hypothesis/eval_source/hypothesis_temporary_module_ad1a1147e61f0def87284e9ecc41a9cea2c90f4d.py FAILED

================================================================= FAILURES ==================================================================
_________________________________________________________ Test_Hypothesis.test_one __________________________________________________________

self = <test_Hypothesis.Test_Hypothesis object at 0x10d8bbeb8>
args = <[AssertionError("assert 0 == 1
 +  where 0 = len([])
 +    where [] = []
 +      where [] = <_ast.Module object at 0x10d8c6a90>.body") raised in repr()] HypothesisProvided object at 0x10d8b3488>

    def test_one(self, args=not_set):
>       return f(self, args)

.hypothesis/eval_source/hypothesis_temporary_module_2de32ff1d5e50f88f5b700adc3c8602d72e8a202.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/core.py:482: in wrapped_test
    (k, convert_to_specifier(v)) for (k, v) in kwargs.items()))
.hypothesis/eval_source/hypothesis_temporary_module_3f7411c21a40bfdc1d4a385395abcf8f35bfc403.py:5: in fixed_dictionaries
    return f(mapping)
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/strategies.py:69: in accept
    arg_string(strategy_definition, args, kwargs_for_repr)
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/reflection.py:283: in arg_string
    bits.append('%s=%s' % (a, unicode_safe_repr(kwargs.pop(a))))
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/compat.py:77: in unicode_safe_repr
    return repr(x)
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/searchstrategy/flatmapped.py:44: in __repr__
    self.expand))
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/reflection.py:265: in get_pretty_function_description
    result = extract_lambda_source(f)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

f = <function <lambda> at 0x10d5a7158>

    def extract_lambda_source(f):
        """Extracts a single lambda expression from the string source. Returns a
        string indicating an unknown body if it gets confused in any way.

        This is not a good function and I am sorry for it. Forgive me my
        sins, oh lord

        """
        args = inspect.getargspec(f).args
        if_confused = 'lambda %s: <unknown>' % (', '.join(args),)
        try:
            source = inspect.getsource(f)
        except IOError:
            return if_confused

        if not isinstance(source, text_type):  # pragma: no branch
            source = source.decode('utf-8')  # pragma: no cover

        try:
            try:
                tree = ast.parse(source)
            except IndentationError:
                source = 'with 0:\n' + source
                tree = ast.parse(source)
        except SyntaxError:
            return if_confused

        all_lambdas = extract_all_lambdas(tree)
        aligned_lambdas = [
            l for l in all_lambdas
            if args_for_lambda_ast(l) == args
        ]
        if len(aligned_lambdas) != 1:
            return if_confused
        lambda_ast = aligned_lambdas[0]
        line_start = lambda_ast.lineno
        column_offset = lambda_ast.col_offset
        source = source[find_offset(source, line_start, column_offset):].strip()

        source = source[source.index('lambda'):]

        for i in hrange(len(source), -1, -1):  # pragma: no branch
            try:
                parsed = ast.parse(source[:i])
>               assert len(parsed.body) == 1
E               assert 0 == 1
E                +  where 0 = len([])
E                +    where [] = []
E                +      where [] = <_ast.Module object at 0x10d8bb320>.body

/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/reflection.py:243: AssertionError
________________________________________________________ Test_Hypothesis.test_three _________________________________________________________

self = <test_Hypothesis.Test_Hypothesis object at 0x10d8c6a58>
args = <[AssertionError("assert 0 == 1
 +  where 0 = len([])
 +    where [] = []
 +      where [] = <_ast.Module object at 0x10d994b00>.body") raised in repr()] HypothesisProvided object at 0x10d8b3588>

    def test_three(self, args=not_set):
>       return f(self, args)

.hypothesis/eval_source/hypothesis_temporary_module_480c901976580cb7f70c301bbce68eeb103f01ba.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/core.py:482: in wrapped_test
    (k, convert_to_specifier(v)) for (k, v) in kwargs.items()))
.hypothesis/eval_source/hypothesis_temporary_module_3f7411c21a40bfdc1d4a385395abcf8f35bfc403.py:5: in fixed_dictionaries
    return f(mapping)
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/strategies.py:69: in accept
    arg_string(strategy_definition, args, kwargs_for_repr)
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/reflection.py:283: in arg_string
    bits.append('%s=%s' % (a, unicode_safe_repr(kwargs.pop(a))))
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/compat.py:77: in unicode_safe_repr
    return repr(x)
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/searchstrategy/flatmapped.py:44: in __repr__
    self.expand))
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/reflection.py:265: in get_pretty_function_description
    result = extract_lambda_source(f)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

f = <function <lambda> at 0x10d5a7158>

    def extract_lambda_source(f):
        """Extracts a single lambda expression from the string source. Returns a
        string indicating an unknown body if it gets confused in any way.

        This is not a good function and I am sorry for it. Forgive me my
        sins, oh lord

        """
        args = inspect.getargspec(f).args
        if_confused = 'lambda %s: <unknown>' % (', '.join(args),)
        try:
            source = inspect.getsource(f)
        except IOError:
            return if_confused

        if not isinstance(source, text_type):  # pragma: no branch
            source = source.decode('utf-8')  # pragma: no cover

        try:
            try:
                tree = ast.parse(source)
            except IndentationError:
                source = 'with 0:\n' + source
                tree = ast.parse(source)
        except SyntaxError:
            return if_confused

        all_lambdas = extract_all_lambdas(tree)
        aligned_lambdas = [
            l for l in all_lambdas
            if args_for_lambda_ast(l) == args
        ]
        if len(aligned_lambdas) != 1:
            return if_confused
        lambda_ast = aligned_lambdas[0]
        line_start = lambda_ast.lineno
        column_offset = lambda_ast.col_offset
        source = source[find_offset(source, line_start, column_offset):].strip()

        source = source[source.index('lambda'):]

        for i in hrange(len(source), -1, -1):  # pragma: no branch
            try:
                parsed = ast.parse(source[:i])
>               assert len(parsed.body) == 1
E               assert 0 == 1
E                +  where 0 = len([])
E                +    where [] = []
E                +      where [] = <_ast.Module object at 0x10d947e80>.body

/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/reflection.py:243: AssertionError
_________________________________________________________ Test_Hypothesis.test_two __________________________________________________________

self = <test_Hypothesis.Test_Hypothesis object at 0x10da53dd8>
args = <[AssertionError("assert 0 == 1
 +  where 0 = len([])
 +    where [] = []
 +      where [] = <_ast.Module object at 0x10d9860f0>.body") raised in repr()] HypothesisProvided object at 0x10d8b3608>

    def test_two(self, args=not_set):
>       return f(self, args)

.hypothesis/eval_source/hypothesis_temporary_module_ad1a1147e61f0def87284e9ecc41a9cea2c90f4d.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/core.py:482: in wrapped_test
    (k, convert_to_specifier(v)) for (k, v) in kwargs.items()))
.hypothesis/eval_source/hypothesis_temporary_module_3f7411c21a40bfdc1d4a385395abcf8f35bfc403.py:5: in fixed_dictionaries
    return f(mapping)
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/strategies.py:69: in accept
    arg_string(strategy_definition, args, kwargs_for_repr)
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/reflection.py:283: in arg_string
    bits.append('%s=%s' % (a, unicode_safe_repr(kwargs.pop(a))))
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/compat.py:77: in unicode_safe_repr
    return repr(x)
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/searchstrategy/flatmapped.py:44: in __repr__
    self.expand))
/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/reflection.py:265: in get_pretty_function_description
    result = extract_lambda_source(f)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

f = <function <lambda> at 0x10d5a7158>

    def extract_lambda_source(f):
        """Extracts a single lambda expression from the string source. Returns a
        string indicating an unknown body if it gets confused in any way.

        This is not a good function and I am sorry for it. Forgive me my
        sins, oh lord

        """
        args = inspect.getargspec(f).args
        if_confused = 'lambda %s: <unknown>' % (', '.join(args),)
        try:
            source = inspect.getsource(f)
        except IOError:
            return if_confused

        if not isinstance(source, text_type):  # pragma: no branch
            source = source.decode('utf-8')  # pragma: no cover

        try:
            try:
                tree = ast.parse(source)
            except IndentationError:
                source = 'with 0:\n' + source
                tree = ast.parse(source)
        except SyntaxError:
            return if_confused

        all_lambdas = extract_all_lambdas(tree)
        aligned_lambdas = [
            l for l in all_lambdas
            if args_for_lambda_ast(l) == args
        ]
        if len(aligned_lambdas) != 1:
            return if_confused
        lambda_ast = aligned_lambdas[0]
        line_start = lambda_ast.lineno
        column_offset = lambda_ast.col_offset
        source = source[find_offset(source, line_start, column_offset):].strip()

        source = source[source.index('lambda'):]

        for i in hrange(len(source), -1, -1):  # pragma: no branch
            try:
                parsed = ast.parse(source[:i])
>               assert len(parsed.body) == 1
E               assert 0 == 1
E                +  where 0 = len([])
E                +    where [] = []
E                +      where [] = <_ast.Module object at 0x10da5f978>.body

/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/hypothesis/internal/reflection.py:243: AssertionError
========================================================= 3 failed in 2.81 seconds ==========================================================
Cems-Mac-Pro:Desktop crushed$
DRMacIver commented 8 years ago

Ouch. Yeah, that function was one of those things that I knew was a bad idea when I was writing it but couldn't resist. I'll investigate what's going on.

If you need a workaround right now, moving the lambda body onto the same line as the lambda definition will avoid this bug.

ckaran commented 8 years ago

-----Original Message----- From: David R. MacIver [mailto:notifications@github.com] Sent: Tuesday, July 21, 2015 10:00 AM To: DRMacIver/hypothesis Cc: Karan, Cem F CIV USARMY ARL (US) Subject: Re: [hypothesis] Failure within extract_lambda_source() AKA, your sins are not forgiven! ;) (#104)

Ouch. Yeah, that function was one of those things that I knew was a bad idea when I was writing it but couldn't resist. I'll investigate what's going on.

If you need a workaround right now, moving the lambda body onto the same line as the lambda definition will avoid this bug.

OK, that workaround will work for me.

Thanks for looking into this!

Thanks, Cem Karan

DRMacIver commented 8 years ago

By the way, the source of the change is that 1.8.0 changed repr for flatmap strategies to include the function definition in the description, which uses this extract lambda source function.

I'm actually not totally clear on why repr is being called on the strategy. That doesn't look quite right, but I might be missing something.

DRMacIver commented 8 years ago

This should be fixed in 1.8.4 (just released).

ckaran commented 8 years ago

-----Original Message----- From: David R. MacIver [mailto:notifications@github.com] Sent: Tuesday, July 21, 2015 6:42 PM To: DRMacIver/hypothesis hypothesis@noreply.github.com Cc: Karan, Cem F CIV USARMY ARL (US) cem.f.karan.civ@mail.mil Subject: Re: [hypothesis] Failure within extract_lambda_source() AKA, your sins are not forgiven! ;) (#104)

This should be fixed in 1.8.4 (just released).

Cool, I'll upgrade immediately!

Thanks, Cem Karan