search

google / pytype

A static type analyzer for Python code
https://google.github.io/pytype
Other
4.78k stars 280 forks source link

Unreplaced NamedType error related to ParamSpec and circular dependencies #1797

Closed tungol closed 1 month ago

tungol commented 1 month ago

In https://github.com/python/typeshed/pull/12745 , I created a circular dependency which caused pytype to generate an Unreplaced NamedType error. It triggers when ParamSpec is used. Interestingly, it only triggers if the file containing the ParamSpec is processed first. If the processing order is reversed, there's no error.

I isolated a minimal reproduction, which is available here: https://github.com/tungol/pytype-error-reproduction

When the script in that repo is run, I get this output:

pytype with files ['bar.pyi', 'foo.pyi']:

Traceback (most recent call last):
  File ".venv312/lib/python3.12/site-packages/pytype/load_pytd.py", line 337, in verify
    mod_ast.Visit(visitors.VerifyLookup(ignore_late_types=True))
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 102, in Visit
    return _Visit(self, visitor, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 120, in _Visit
    return _VisitNode(node, visitor, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 171, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 171, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 171, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 194, in _VisitNode
    status = visitor.Enter(node, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/base_visitor.py", line 211, in Enter
    return self.enter_functions[node.__class__.__name__](
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/visitors.py", line 258, in EnterNamedType
    raise ValueError(f"Unreplaced NamedType: {node.name!r}")
ValueError: Unreplaced NamedType: 'typing.Callable'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venv312/lib/python3.12/site-packages/pytype/load_pytd.py", line 710, in finish_and_verify_ast
    self._resolver.verify(mod_ast)
  File ".venv312/lib/python3.12/site-packages/pytype/load_pytd.py", line 340, in verify
    raise BadDependencyError(str(e), name) from e
pytype.load_pytd.BadDependencyError: Unreplaced NamedType: 'typing.Callable', referenced from 'foo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".venv312/lib/python3.12/site-packages/pytype/load_pytd.py", line 337, in verify
    mod_ast.Visit(visitors.VerifyLookup(ignore_late_types=True))
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 102, in Visit
    return _Visit(self, visitor, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 120, in _Visit
    return _VisitNode(node, visitor, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 171, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 171, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 212, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 171, in _VisitNode
    new_child = _VisitNode(child, visitor, *args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/parse/node.py", line 194, in _VisitNode
    status = visitor.Enter(node, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/base_visitor.py", line 211, in Enter
    return self.enter_functions[node.__class__.__name__](
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv312/lib/python3.12/site-packages/pytype/pytd/visitors.py", line 258, in EnterNamedType
    raise ValueError(f"Unreplaced NamedType: {node.name!r}")
ValueError: Unreplaced NamedType: '_P.args'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./pytype_test.py", line 33, in main
    loader.finish_and_verify_ast(ast)
  File ".venv312/lib/python3.12/site-packages/pytype/load_pytd.py", line 731, in finish_and_verify_ast
    self._resolver.verify(mod_ast)
  File ".venv312/lib/python3.12/site-packages/pytype/load_pytd.py", line 340, in verify
    raise BadDependencyError(str(e), name) from e
pytype.load_pytd.BadDependencyError: Unreplaced NamedType: '_P.args', referenced from 'foo'

Ran pytype with 2 pyis, got 1 errors.

foo.pyi (3.12): pytype.load_pytd.BadDependencyError: Unreplaced NamedType: '_P.args', referenced from 'foo'
pytype with files ['foo.pyi', 'bar.pyi']:
Ran pytype with 2 pyis, got 0 errors.
pytype with files ['foo.pyi']:
Ran pytype with 1 pyis, got 0 errors.
pytype with files ['bar.pyi']:
Ran pytype with 1 pyis, got 0 errors.

I wasn't able to reproduce the error with .py files instead if .pyi files, but I also didn't try too hard, so I don't know for sure if that's a critical part of the error or not.

frigus02 commented 1 month ago

Thanks for reporting. I'm going to take a look today or tomorrow. I hope that's easy to fix.

frigus02 commented 1 month ago

Thanks for the tiny example. I can easily reproduce as described.

I wonder if this is an unintended use of the loader API. Moving loader = load_pytd.create_loader(options, []) inside the for loop fixes the issue.

I still want to debug why exactly this error is happening, though. I don't think it should.

frigus02 commented 1 month ago

Argh. I tried running the typeshed tests with a new loader for each file. Took way too long. Then tried a new loader for each file but at least caching builtins and typing base modules. Still way too slow. I guess we will have to fix this in pytype somehow.

frigus02 commented 1 month ago

I might have a fix: #1804. I need to do some more testing tomorrow to ensure that this doesn't cause a regression elsewhere. But it looks promising.

frigus02 commented 1 month ago

We just published https://pypi.org/project/pytype/2024.10.11/

I tested that with your PR on typeshed and confirmed that the tests pass.

tungol commented 1 month ago

Nice! I appreciate it.