HypothesisWorks / hypothesis

Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
https://hypothesis.works
Other
7.58k stars 586 forks source link

attrs type annotation support in builds() or from_type() #4098

Open Hnasar opened 2 months ago

Hnasar commented 2 months ago

Modern attrs supports annotations just like dataclasses, but hypothesis doesn't handle them as well. (This is a mix between a bug report and feature request 😄 )

For example,

  1. if there is a required attribute, hypothesis isn't aware how to provide it.
  2. And if there is an optional attribute, hypothesis only provides the default rather than generating a strategy based on the annotation.

1) error from required attribute

import dataclasses

@attrs.define
class FooAttrs:
    age: int
    name: str = "bob"

@dataclasses.dataclass
class FooDataclass:
    age: int
    name: str = "bob"

@given(foo_attrs=st.builds(FooAttrs, age=...), foo_dataclass=st.builds(FooDataclass, age=...))
def test_foo_required(
    foo_attrs: FooAttrs,
    foo_dataclass: FooDataclass
) -> None:
    pass

results in an unexpected error — I expect it would handle it the same as dataclass.

        # Better to give a meaningful error here than an opaque "could not draw"
        # when we try to get a value but have lost track of where this was created.
        if strat.is_empty:
>           raise ResolutionFailed(
                "Cannot infer a strategy from the default, validator, type, or "
                f"converter for attribute={attrib!r} of class={target!r}"
            )
E           hypothesis.errors.ResolutionFailed: Cannot infer a strategy from the default, validator, type, or converter for attribute=Attribute(name='age', default=NOTHING, validator=None, repr=True, eq=True, eq_key=None, order=True, order_key=None, hash=None, init=True, metadata=mappingproxy({}), type='int', converter=None, kw_only=False, inherited=False, on_setattr=None, alias='age') of class=<class 'test_mypy_caches.FooAttrs'>

2) and if I add a default to age:

@attrs.define
class FooAttrs:
    age: int = 0
    name: str = "bob"

@dataclasses.dataclass
class FooDataclass:
    age: int = 0
    name: str = "bob"

then this results in always using age=0 although I expect that it would match the behavior of dataclass. (Shown using pytest's --hypothesis-verbosity=debug)

4 bytes [[0, 137]] -> Status.VALID, 
Trying example: test_foo_required(
    foo_attrs=FooAttrs(age=0),
    foo_dataclass=FooDataclass(age=-88),
)
4 bytes [[0, 66225]] -> Status.VALID, 
0 bytes [] -> Status.INVALID, 
Trying example: test_foo_required(
    foo_attrs=FooAttrs(age=0),
    foo_dataclass=FooDataclass(age=10196),
)

Workaround

# one of:
st.builds(FooAttrs, age=st.from_type(attrs.fields(attrs.resolve_types(FooAttrs)).age.type))
st.builds(FooAttrs, age=st.from_type(int))
st.builds(FooAttrs, age=st.integers())

version info

This is with hypothesis 6.108.2, attrs 23.1.0, and python3.11

Discussion

I see that https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.builds has a blurb about attrs

If the callable is a class defined with attrs, missing required arguments will be inferred from the attribute on a best-effort basis, e.g. by checking attrs standard validators. Dataclasses are handled natively by the inference from type hints.

but these behaviors were still unexpected and the workaround wasn't clear.

Zac-HD commented 2 months ago

Both of these are because we're not currently inferring from the type; makes sense to fix that 🙂