A possible change to match_class_typing decorator type hints

[ramblehead]

The following code that loads statically-typed JSON works as expected:

# test.py

import json
from typing import TypedDict

config_json_str = """
{
  "id": 42,
  "name": "baz"
}
"""

class Config(TypedDict):
    id: int
    name: str

def load() -> Config:
    return json.loads(config_json_str)

# conf_id type is Config
conf = load()

# conf_id type is int
conf_id = conf["id"]

# conf_id type is str
conf_name = conf["name"]

print(f"Conf vals: {conf_id}, {conf_name}")

The following key usage error is detected by pyright at "compile time" and by python at runtime.

# runtime: KeyError: 'babayka'
# pyright check: Type of "conf_name" is unknown [reportUnknownVariableType]
conf_babayka = conf["babayka"]

The following config_json_str "name" key error would not be detected at "compile time" and at runtime it will only be detected when trying to access non-existing "name" key in conf dict:

config_json_str = """
{
  "id": 42,
  "babayka_name": "baz"
}
"""

strongtyping can bring config keys and shape errors to load() stage or (better) to test runners, which is awesome :sunglasses: For example:

# test_typing_issues.py

import json
from typing import TypedDict

from strongtyping.strong_typing import match_class_typing, match_typing

config_json_str = """
{
  "id": 42,
  "name": "baz"
}
"""

@match_class_typing
class Config(TypedDict):
    id: int
    name: str

@match_typing
def validate_config(config: Config) -> Config:
    return config

def load() -> Config:
    return validate_config(json.loads(config_json_str))

# conf_id type is Config
conf = load()

# conf_id type is str
conf_id = conf["id"]

# conf_id type is str
conf_name = conf["name"]

print(f"Conf vals: {conf_id}, {conf_name}")

The above test_typing_issues.py works as expected at runtime, pyright however produces the following errors:

<...>/test_typing_issues.py
  <...>/src/python/cdaq_proto/test_typing_issues.py:21:29 - warning: Expected type expression but received "match_class_typing" (reportGeneralTypeIssues)
  <...>/src/python/cdaq_proto/test_typing_issues.py:21:40 - warning: Expected type expression but received "match_class_typing" (reportGeneralTypeIssues)
  <...>/src/python/cdaq_proto/test_typing_issues.py:21:21 - error: Type of parameter "config" is unknown (reportUnknownParameterType)
  <...>/src/python/cdaq_proto/test_typing_issues.py:22:12 - warning: Return type is unknown (reportUnknownVariableType)
  <...>/src/python/cdaq_proto/test_typing_issues.py:21:40 - warning: Declared return type is unknown (reportUnknownVariableType)
  <...>/src/python/cdaq_proto/test_typing_issues.py:30:15 - warning: Expected type expression but received "match_class_typing" (reportGeneralTypeIssues)
  <...>/src/python/cdaq_proto/test_typing_issues.py:30:15 - warning: Declared return type is unknown (reportUnknownVariableType)
  <...>/src/python/cdaq_proto/test_typing_issues.py:35:1 - warning: Type of "conf" is unknown (reportUnknownVariableType)
  <...>/src/python/cdaq_proto/test_typing_issues.py:38:1 - warning: Type of "conf_id" is unknown (reportUnknownVariableType)
  <...>/src/python/cdaq_proto/test_typing_issues.py:41:1 - warning: Type of "conf_name" is unknown (reportUnknownVariableType)
1 error, 9 warnings, 0 informations

A hackerish way to resolve the above typing issues at user side could separating strongtyping-decorated classes, explicit casting and disabling the issue of "variable used as type":

# test_hackerish.py

import json
from typing import TypedDict, cast

from strongtyping.strong_typing import match_class_typing, match_typing

config_json_str = """
{
  "id": 42,
  "name": "baz"
}
"""

class Config(TypedDict):
    id: int
    name: str

ConfigStrong = match_class_typing(Config)

@match_typing
def validate_config(config: ConfigStrong) -> Config:  # type: ignore reportGeneralTypeIssues
    return cast(Config, config)

def load() -> Config:
    return validate_config(json.loads(config_json_str))

# conf_id type is Config
conf = load()

# conf_id type is str
conf_id = conf["id"]

# conf_id type is str
conf_name = conf["name"]

print(f"Conf vals: {conf_id}, {conf_name}")

In my understanding, to eliminate the above typing issues without user-side hacking, strongtyping decorators type hints should be changed to identity functions. For example:

# strongtyping-stubs/strong_typing.pyi

# Remove original "class match_class_typing: ..."
# ...

from typing import TypeVar

T = TypeVar("T")

def match_class_typing(cls: T) -> T: ...

# ...

This fix should eliminate typing issues in the above "test_typing_issues.py" example. The "class match_class_typing: ..." could be defined under a different name such as "class MatchClassTyping: ...". If library users need access to MatchClassTyping class members, they can just cast their "match_class_typing" decorated class to MatchClassTyping.

Probably, the above is applicable to other strongtyping decorators...

[FelixTheC]

Wow really cool. I will definitly have a look into it to fix this.

[FelixTheC]

@ramblehead If I would have a possible fix could I prepare the package for you without publishing so that you can confirm my changes??

[ramblehead]

Certainly! If you push it to another branch, I could check/test it to the best of my knowledge.

I think, this library is a great idea with many potential use cases. I am currently trying to use it for json configuration loading as shown in my example above. In the future, it should be possible to do "on-the-fly" conversion of jtd to TypedDict for nice DX and guided refactoring, and also use strongtyping to perform run-time checks on untrusted json data.

[FelixTheC]

Please checkout this branch you can also install it with poetry

[FelixTheC]

I merged your PR, and it seems to fit with Pyright, but we need to do some more work to make it valid for Mypy. Which IDE you're using??

[FelixTheC]

I made it now also valid for Mypy I used your example.

import json
from typing import TypedDict

from strongtyping.strong_typing import match_class_typing, match_typing
from strongtyping.typed_namedtuple import typed_namedtuple

config_json_str = """
{
  "id": 42,
  "name": "baz"
}
"""

@match_class_typing
class Config(TypedDict):
    id: int
    name: str

@match_typing
def validate_config(config: Config) -> Config:
    return config

def load() -> Config:
    return validate_config(json.loads(config_json_str))

Cat = typed_namedtuple("Cat", [("allow", bool), ("block", bool)])

def main():
    # conf_id type is Config
    conf = load()

    # conf_id type is str
    conf_id = conf["id"]

    # conf_id type is str
    conf_name = conf["name"]

    print(f"Conf vals: {conf_id}, {conf_name}")

if __name__ == "__main__":
    main()

[ramblehead]

Agreed - more work is always needed :smile: I am on Emacs with lsp-mode at pyright and ruff. Both pyright and ruff (with rather strict settings) are complaining at the code in abandance e.g.:

I will try to find some time this week to configure mypy as well. For my current use case (config load with run-time check) the last version seems to be good enough :rocket:

[FelixTheC]

I will then create a new release. And open a new issues/branch to fix the other type annotation issues??

[ramblehead]

Thanks for the quick fixes. Great job!

I think, this issue can be closed - lets keep one squiggly line per issue...