Arguments to decorators as dependencies of the decorated function

rasmuse commented 4 years ago

Forgive me (and please let me know) if I'm asking a question that is out of scope for this issue tracker. I'm trying to use mypy to do things it was not necessarily designed to do.

Specifically, I'm trying to use the fine-grained dependency mapping (see, e.g., mypy/server/deps.py) to find the source code that is needed for a given function. Concretely I'm running mypy --logical-deps example.py and then inspecting the dependency graphs generated in .mypy_cache/3.6/example.deps.json.

It seems to work pretty well for my purposes except that arguments to decorators are not listed as dependencies of the decorated function, e.g., in the following example

some_var = 1
@my_decorator(some_var)
def my_func(x):
    return x

Here, for my purpose, some_var is a dependency of my_func in the sense that it is needed to run my_decorator which returns the decorator for my_func. However, mypy does not list some_var as a dependency of my_func. If I understand the type checking correctly, this makes sense because it is not a dependency in terms of type checking; the type of some_var is relevant to type check the call to my_decorator, and the return type of my_decorator is relevant to type check the decoration, but there is no direct connection between some_var and my_func, right?

My question is if I have understood this correctly. This issue can be closed if someone knows this well enough to say "forget it kid, mypy is never going to solve your problem". My hope of course is that someone will say "yeah, that actually is a dependency, and here is the patch we need to write".

Thank you for your time.

JelleZijlstra commented 4 years ago

I'd think that logically this can be a dependency, because if my_decorator is overloaded or contains typevars, the type of some_var can affect the type of my_func.

msullivan commented 4 years ago

I think your analysis is all correct.

It doesn't need to be a dependency from a typing perspective because the module toplevel depends on some_var, so if its type changes, then it will get rechecked, and then the change in the function type should trigger a function recheck.

I think it would definitely make sense for it to be a logical dependency, if you want to submit a PR for it. I think maybe the best approach is to update the current scope and do another visit of the decorator. (In visit_decorator in deps.py)

rasmuse commented 4 years ago

Thanks for the input!

I might write a PR for this, but first have to consider whether it's the best solution for my current problem. Meanwhile a couple of questions:

I have not understood what a logical dependency is after searching specifically for it in the source and the docs. There is some explanation in the PR where it was implemented https://github.com/python/mypy/pull/5367 but honestly I don't understand what it means to "reflect propagation of type (im)precision". Can someone please explain and/or point to relevant documentation?
Is it a problem that the decorator syntax @my_decorator(some_var) etc would introduce a logical dependency but not the equivalent my_function = my_decorator(some_var)(my_function)? The latter case I suppose is much harder to analyze.

msullivan commented 4 years ago

Logical dependencies are a somewhat fuzzy concept that we built to help with some internal reporting tools that try to track where Any types can come from. So the main idea I think was that a has a logical dependency on b if Any types in b can lead to Any showing up in A.

I'm not really worried about that. That isn't a common idiom and I doubt mypy supports it well.

python / mypy

Arguments to decorators as dependencies of the decorated function #7494