Add type hints for @typechecked

[dbarnett]

Adds a py.typed marker file to indicate pytypes has type hints (see PEP 561), and adds type hints for @typechecked. These type hints are added in a separate pyi file to help with compatibility for python 3.3 & 3.4, since inline type annotations are a syntax error for those versions.

~The fixes in typechecker.py were for issues flagged by the typechecker and shouldn't change the behavior of the code at all. They may not be strictly needed since type hint resolution order is supposed to check .pyi files before their associated .py files.~ EDIT: Dropped the type checker fixes. They were more trouble than I expected and probably not needed.

Fixes #74.

[Stewori]

This breaks too many tests. Please try if reverting the lines I commented fixes this. One would think it cannot be caused by anything else as the asserts etc are not invasive. But then there was the recent incident with the blank line breaking a test...

[dbarnett]

This breaks too many tests. Please try if reverting the lines I commented fixes this. One would think it cannot be caused by anything else as the asserts etc are not invasive. But then there was the recent incident with the blank line breaking a test...

Yeah, you're right, sorry for the noise. I think I'm going to drop all the little fixes from this PR since any remaining improvements are pretty minor by themselves.

Besides that, I'm also updating the pyi file to be more complete and include types for no_type_check. I don't understand from the docs I've read what behavior things will give if something's defined in .py but not in .pyi. I worry that it'll start complaining "module 'pytypes.typechecker' has no attribute 'foo'" where it originally had more self-explanatory "No such module pytypes". I'm not sure if that's going to be a pain to keep up-to-date, but I suspect it won't matter much either way, if you're game to push to pypi and see if it's an improvement in practice.

[Stewori]

Would it work to make this a pytypes.pyi rather than a typechecker.pyi? pytypes imports all relevant bits and makes them available as pytypes.xxx, e.g. pytypes.typechecked. This would allow for a clean distinction: everything directly available in pytypes would have type-annotations (ideally, one day) while other stuff is considered internal. Also, this would allow to bundle all type info in one place, notably also for pytypes.is_subtype and pytypes.is_of_type.

Or does the static typechecker strictly require the pyi to target the location where the functions are actually implemented?

[dbarnett]

Yeah, that's fair, I agree that's probably the way to go. Let me look into that and update you when it's ready for another look.

[Stewori]

I have second thoughts on the pytypes.pyi idea. IIRC stubfiles are supposed to match the functions and classes as implemented in a module, not imported stuff. Stubfiles are really meant to augment code in static sense while imports and aliases are runtime stuff. So, probably it's best to proceed according to your initial proposal.

At first glance I thought this could solve the issue with annotating functions from type_util, e.g. is_subtype (alias for type_util._issubclass). Traditionally, type info for these functions causes infinite runtime typecheck recursion in profiler mode. I guess the best solution would be to go from from type_util import _issubclass as is_subtype to

from type_util import _issubclass

def is_subtype(...): #with type annotation
    return _issubclass(...)

but anyway, that's beyond this PR. Just explaining my reasoning that led to the idea with pytypes.pyi.

[Stewori]

I suppose there is a way to let mypy validate whether the pyi file still fits to the corresponding py file. E.g. whether all functions and classes are present, and if the signatures match. If you have the right command at hand, I would like to note it down for pre-release checks to avoid a change in the py file was forgotten to be applied in the pyi.

[dbarnett]

I suppose there is a way to let mypy validate whether the pyi file still fits to the corresponding py file.

Looks like no per python/mypy#5028. What I'd done is used stubgen to create a skeleton pyi file and then modified it to add the desired type hints. I could probably formalize it with a quick script to generate and patch a fresh pyi file based on the latest .py file changes. There doesn't seem to be a good automated way to detect discrepancies, though.

[Stewori]

Isn't there a (simple) way to get mypy in case of an invalid stubfile produce these little "no such attribute" errors you mentioned? I assume this would not catch every possible incompatibility but it would be a handy consistency check.

There doesn't seem to be a good automated way to detect discrepancies, though.

Maybe we should consider to add such kind of thing to pytypes. It's intended to host useful utilities for typing, actually. I think stubfile_manager already does some portion of this. I think it would error if the pyi defines something that is not in the py file (including methods and other OOP constructs and nested classes which were especially painful to get right I remember). However, it does not check the "totality" aspect ATM. Also, I am not fully sure what could and should be done in terms of signature checking. I guess the result of inspect.get(full)argspec should be similar between every py and pyi defined function and method. Or is that too restrictive?

[dbarnett]

Isn't there a (simple) way to get mypy in case of an invalid stubfile produce these little "no such attribute" errors you mentioned?

The "no such attribute" errors I mentioned were from calling code referencing pytypes code that wasn't in the pyi. The unsolved problem is comparing the .py to the .pyi and noticing discrepancies like missing functions, wrong number of params, etc.

Maybe we should consider to add such kind of thing to pytypes. It's intended to host useful utilities for typing, actually. I think stubfile_manager already does some portion of this.

Good idea! I filed issue #100 for this. I hadn't seen anything about stubfile_manager yet and can't find any usage notes on it. Sounds like I should look into what it can do.

[Stewori]

I hadn't seen anything about stubfile_manager yet and can't find any usage notes on it. Sounds like I should look into what it can do.

It's just internal and hardly documented. It locates stubfiles and attaches the types therein to the members of the original module. It is the motor of @annotations. Unfortunately it predates PEP 561 and only locates stubfiles "naively" by the logic and heuristics that were common before PEP 561. It's a long-standing todo to implement PEP 561 here properly.

[Stewori]

Sorry to bother again. I already thought it could be merged, then the two cleanup comments occurred to me...

[dbarnett]

K, should be 100% good to go now. Take a look.

[Stewori]

Thanks for working this through!