Evaluate arg types of TypeDicts' Required and NotRequired types instead of origin types

[dimko]

Is your feature request related to a problem? Please describe. We're using TypeDict along with @match_class_typing of strongtyping module to apply type validations for api schemas. We often have the need to define a TypedDict with some keys that are required and others that are optional and can be missing. Previously the only way to define such a TypedDict was to declare one TypedDict with one total=True and then inherit it from another TypedDict with total=False

example

class _Example(TypedDict): # implicitly total=True foo: str

class Example(_Example, total=False): bar: int

Recently TypeDicts started supporting Required and NotRequired types (from typing or typing_extensions modules) for their attributes. source https://peps.python.org/pep-0655/#specification

So a TypeDict class can be easily defined as:

class Example(TypedDict, total=False): foo: Required[str] bar: int

Required[type] and NotRequired[type] have origin type Required|NotRequired and e.g str, int, dict, list, etc as their type args. So strongtyping complains and gives a type mismatch error in case a key is of type str for example, but has been defined as Required[str]

It would be great if strongtyping could support getting type arguments in case of Required and Not Required "types" instead of the origin type

Describe the solution you'd like The simplest solution I can think of is adding an extra if clause in strong_typing_utils.py to handle Required and NotRequired e.g

      if any(x in origin_name for x in ["required", "notrequired"]):
          if get_origin(get_args(type_of)[0]):
              type_of = get_origin(get_args(type_of)[0])
          return isinstance(argument, get_args(type_of))

def check_type(argument, type_of, mro=False, **kwargs):
    from strongtyping.types import IterValidator, Validator

    if checking_typing_generator(argument, type_of):
        # generator will be exhausted when we check it, so we return it without any checking
        return argument

    check_result = True
    if type_of is not None:
        origin, origin_name = get_origins(type_of)
        origin_name = origin_name.lower()

     if any(x in origin_name for x in ["required", "notrequired"]):
            if get_origin(get_args(type_of)[0]):
                type_of = get_origin(get_args(type_of)[0])
            return isinstance(argument, get_args(type_of))

        if "new_type" in origin_name:
            type_of = type_of.__supertype__
            origin, origin_name = get_origins(type_of)
            origin_name = origin_name.lower()
            .
            ..
            ...
            ....

Describe alternatives you've considered Ideally, we could define in the decorator specific origins that should be "ignored" and take into account their args according to the following https://peps.python.org/pep-0655/#interaction-with-get-origin-and-get-args The user could define as a string the original name of the type e.g Required and this would get lowercased for comparison reasons in the utility function.

Additional context

[FelixTheC]

Thx for your issue and your good explaination.

[FelixTheC]

Which Python Version??

[dimko]

Hey @FelixTheC, thanks for the response. I am using Python 3.10.2 with typing_extensions module (version 4.3.0)to gain access to Required|NotRequired. I've also tried with Python 3.11 where I could import Required|NotRequired from typing module

[dimko]

Btw, it also seems that when calculating the required keys of a TypeDict class that has total=True (the default value), if there are keys specified as NotRequired[...] they are not ignored. I partially monkey patched it by adding the following piece of code, but I believe more cases will arise

def checking_typing_typedict_values(args: dict, required_types: dict, total: bool):
    if total:
        for k, v in list(required_types.items()):
            if any(x in str(v) for x in ["typing.Required", "typing.NotRequired"]):
                required_types.pop(k)
        return all(check_type(args[key], val) for key, val in required_types.items())
    fields_to_check = {key: val for key, val in required_types.items() if key in args}
    return all(check_type(args[key], val) for key, val in fields_to_check.items())

[FelixTheC]

Thx for your snippet and your info. But for clarification.

    class MyDict(TypedDict):
        sales: int
        country: str
        product_codes: List[str]
        manger: NotRequired[str]

Both are valid for the TypedDict above

MyDict({"sales": 10, "country": "Hogwards", "product_codes": ["1", "2", "3"]})
MyDict({"sales": 10, "country": "Hogwards", "product_codes": ["1", "2", "3"], "manager": "Dumbledor"})

and having this

    class MyDict(TypedDict, total=False):
        sales: int
        country: str
        product_codes: List[str]
        manger: Required[str]

the minimum we need is

MyDict({"manager": "Snape"})

[dimko]

Thanks for the quick reply.

OK it might be then that the type I use is a more complicated one, something like NotRequired[dict[AnotherTypedDictClass]]

For some reason I get that the NotRequired key is mandatory 🤔

Anyway, will try and keep this one clean and maybe open a more detailed issue for this

[FelixTheC]

Sry I mean my as a question. Can you please write me a class or several once that I can use in my tests.

And if it would be ok for you for each case a test that I really understand what should Happen when.

[dimko]

Sorry closed it by mistake 😅 yes will post an example. Thanks a lot.

[FelixTheC]

@dimko can you please check typeddict tests they're following the implementation specification PEP-655 and should fit your needs, if so please tell me so that I can create a new release soonish

[dimko]

Hey @FelixTheC. Is it possible to also check in tests how it behaves when you use another TypedDict class as the arg of Required|NotRequired e.g NotRequired[dict[FantasyMovie]] or doesn't it not make any difference? I believe it covers our scenarios. Many thanks for this 🙂

[FelixTheC]

I expect that this won't make a difference but I will add a test case to be 100% sure.

[FelixTheC]

@dimko

def test_typeddict_with_required_and_not_required_and_sub_typeddict():
    from typing import TypedDict

    @match_class_typing
    class Movie(TypedDict):
        title: str
        year: NotRequired[int]

    @match_class_typing
    class Additional(TypedDict):
        name: str
        val: NotRequired[str]

    @match_class_typing
    class Regisseur(TypedDict):
        name: str
        movie: Required[dict[Movie]]
        year: Required[int]
        info: NotRequired[dict[Additional]]

    assert Regisseur(name="Alfonso Cuarón", movie=Movie(title="Hallow"), year=2004)

    with pytest.raises(TypeMisMatch):
        Regisseur(name="Alfonso Cuarón", movie=Movie, year=2004)

    with pytest.raises(TypeMisMatch):
        Regisseur(name="Alfonso Cuarón", year=2004)

everything is green

[dimko]

Hey thanks for this. Just one question cause I am a bit confused about how I should be using it. According to this https://peps.python.org/pep-0655/#usage-in-python-3-11 TypedDict and (Not)Required should be imported from typing_extensions for python <3.11. I see in the tests that TypedDict is imported from typing and the other 2 from typing_extensions. When I do this I get an error and returned dictionaries are always {}. Am I missing something?

[FelixTheC]

Hmm thats a good question I use it from typing as it is allready there. Are you using in your project both from typing-extensions??

[dimko]

For the python 3.10 checks I did yes. Both from typing_extensions, otherwise I was getting an error (will post more details on the error later). I moved to 3.11 2 days ago though, to prepare for any possible migrations that might be needed. In 3.11 I import everything from typing.

[FelixTheC]

I Added support for TypedDict from typing-extension. But I'm not sure how fast I can test against Python3.11 with TypedDict from typing-extensions

[dimko]

No worries. Thanks for the help. Maybe merge and close this and we can test again, fully, once 3.11 is officially released?

[FelixTheC]

looks like 2022-10-24

[FelixTheC]

ok I will merge it and create a new release

[FelixTheC]

@dimko new release is 3.10.3_20220810