Closed engnatha closed 7 months ago
What is going on here is that if the type annotation for a field references an undefined type, an exception will be thrown. While the exception message may be a bit opaque, the exception itself does not seem unreasonable to me in this case. The type annotation does, in fact, reference an undefined type — this is, arguably, a syntax error.
Were the annotation to reference any resolvable type, using metadata["marshmallow_field"]
to control specify the schema field would work.
I'm inclined to say the described behavior is expected and not a bug.
This may be a stupid question, but: What is the purpose of using a string literal that references an undefined type in the field's type annotation?
I.e. what benefit results from writing
@dataclass
class Demo
a: "NotAType" = field(metadata={"marshmallow_field": ...})
rather than, e.g.
from typing import Any
@dataclass
class Demo
a: Any = field(metadata={"marshmallow_field": ...})
That's an excellent explanation of what's going on and was the conclusion that I came to. The benefit is that there are certain annotations that only exist in type stubs. The example I wrote was contrived, but my specific use case involved using mypy-protobuf. This generates type annotations for protobuf files that let you do really helpful things like autocomplete in editors.
The case where this came up was using their enum implementation. You get to annotate things like my_var: 'my_pb2.MyEnum.V'
which means my_var
should only take on values found in MyEnum
. At run time, MyEnum
values are just integers so type hinting as int
isn't very useful. Tools like mypy and pyright (which we use) do know how to interpret these annotations, correlate them to the stub file, and do better static analysis than if we had just said my_var: int
.
So, I'm on board with you that we can't find a type at run time. After all, that's exactly how type stubs work. The part that I think would be useful to change is that it raises an error even if the author has supplied the marshmallow information in the field
metadata section. It seems that's all marshmallow_dataclass needs to know. I was hoping by supplying that metadata information, we could get around the specific type of the field not being able to be determined since it doesn't really need to be determined at that point.
@engnatha Thank you! I think I get it now. If I understand correctly, a workaround would be something like:
if TYPE_CHECKING:
from my_stubs import MyType
else:
MyType = Any # or some other more specific concrete type
@dataclass
class Demo
a: MyType = field(metadata={"marshmallow_field": ...})
I think you're right: if there is a custom marshmallow_field
it would be possible to skip getting the type hint for that field.
In the general case, however, we may still need type hints for other fields in the dataclass. Is it possible to get those without computing (as we do now) the type hints for all the fields in one go?
I expect that will work out. I'll give it a try. The structure of the code I linked made it seem like it would be possible to do some inspection before to see if there's a need for fetching the type of the field. The order seemed to be
Do you think it's feasible to do
If there's no issues with the remainder of the marshmallow code (since dataclasses on their own work happily with type stub annotations), then I could see that change being more extensible for users.
On python 3.8 (since haven't upgraded yet), I had to do the following
from typing_extensions import TypeAlias
if typing.TYPE_CHECKING:
_MyType: TypeAlias = 'my_pb2.MyEnum.V'
else:
_MyType: TypeAlias = 'Any'
Without the TypeAlias
, pyright complains that we're trying to use a variable as a type annotation.
2. Infer fields for the remaining types
The issue here is that, currently, we get the annotations for all of the fields in the dataclass using typing.get_type_hints in one fell swoop. (And get_type_hints
raises the exception in question if any single field of the dataclass has an unresolvable type annotation.)
I'm fairly certain that get_type_hints
can not be applied to individual class attributes — only to an entire class, in bulk. (This is, at least in part, because the annotations for class attributes are stored in an extra __attributes__
field of the class, not on the individual attributes.)
If there is some other clean way to resolve the type hints for individual fields (or a subset of fields) of a dataclass — and there may well be — I don't currently know what it is.
We could, I suppose, skip the call to get_type_hints
in the special case that all the fields have an explicit marshmallow_field
set. But that's a pretty limited use case. Doing so would likely cause more confusion than it solves.
Sorry for the delay on this. I agree with your take on the last point; I don't want the alternative to be custom defining all the types as soon as we have to define one.
I think it's a fair assessment that this is a limitation of the type hinting library. The method above is an acceptable workaround when this pops up, so I'm going to close. Maybe the future of get_type_hints
will be more flexible.
When using python 3.9 and above, users are not able to use type stub-only fields. The docs indicate one may use
marshmallow_field
in the metadata to control the marshmallow type that a field should be in the marshmallow schema. However, https://github.com/lovasoa/marshmallow_dataclass/blob/d6396c18470582a4fe5f0f2bd29ac012da4f0f1f/marshmallow_dataclass/__init__.py#L433-L434 indicates thattyping.get_type_hints
is called prior to inspecting for a custommarshmallow_field
definition.Below is an example that minimally reproduces this behavior with version 8.6.0.
Instead of getting this error, I would expect the tooling to gracefully skip over the missing object definition (as would be the case in a stub-only object) in favor of the existing override.