Python 3.7 support

[Stewori]

Compatibility with the new upcoming typing module is not yet established. I'm working on this as far as time allows, help welcome. https://github.com/Stewori/pytypes/commit/928fd6257272656c346e91fe433dd4e0fc08cff7 was the first milestone of this process, enabling the test suite to complete without a crash (not speaking of failing tests). Now, still most tests are failing. Work to do...

[mitar]

How is this going? Is there some branch to look the progress at?

One suggestion could be that you use typing_inspect instead of manually accessing properties. In this way you might have easier time updating to new Python version.

[Stewori]

I was bound by another project and still am. When I started pytypes I did not expect breaks in the typing module of this magnitude. Will Python 3.8 break everything again? Maybe it's pointless to use typing features while typing is provisional. It is demotivating to be not able to move the project forward because all time goes into fixing breakages. That said, I think when I investigated this I found that the main issues are in type_util generic handling code, i.e. in the functions:

_find_base_with_origin
_select_Generic_superclass_parameters
_find_Generic_super_origin
_issubclass_Generic

I think these would require some isolated tests in order to properly fix the divergence. I doubt that typing_inspect features this functionality but it has been a while since I looked at it. Correct me if I'm wrong. Also, typing_inspect is not Python 2.7 compatible, which was traditionally a goal of pytypes (e.g. for Jython support). That's the main reason why I never took on typing_inspect. Also, pytypes predates typing_inspect and most stuff was already solved before typing_inspect arose. It would have been additional work to switch the utilities again.

[Stewori]

I think the best approach to get test cases for the named methods would be to profile their input/output in Python 3.6 using some decorator or so during tests. Then compare it with Python 3.7 output.

[mitar]

Will Python 3.8 break everything again?

From what I read around, the answer is no. :-) I don't think there is any PEP around changing anything more. Also, I think the plan is/was to make things not provisional anymore in 3.7. I think they are keeping it provisional for now just to be on the safe side.

So yea, finally things are stabilizing. :-)

[Stewori]

With "just" 24 skips, regular testing of those features that already work on Python 3.7 just launched. Maybe a Christmas calendar would be a suitable motivation to count-down the remaining breakages.

[Stewori]

Was falsely auto-closed by github.

[Stewori]

Just 3 remaining skips. Python 3.7 support is near...

[mitar]

Sending you a huge hug to encourage finalizing the Python 3.7/8 support. :-)

[Stewori]

Sorry, the last remaining skip turned out to be non-trivial and is rather hard to fix. I cannot tell when I will get it done.

[mitar]

Oh, what is the issue?

[Stewori]

Various failures in test_unparameterized.

[dbarnett]

Would more hands on this help, or too involved for that?

[mitar]

A gentle nudge here. Python 3.6 support will slowly end this year.

[aronszanto]

Stefan, thanks so much for your fantastic work on this project. We use pytypes throughout our codebase because it is the only typechecker that is able to properly handle arbitrary generics, which makes it absolutely invaluable in complex type environments. It's not lost on us that this utility is the result of an immense amount of hard work- type systems are wildly tricky to get right, especially in Python. I am sure that there are many others in this position who find pytypes instrumental to their technology and are similarly grateful to you.

Like many organizations, we're slowly migrating off of python3.6, as recommended by the deprecation schedule posted by @mitar. It would be extremely helpful to understand what your thinking is as far as support for 3.7 and beyond. I know that what seemed a couple of years ago as a tractable update turned into something much more complex given the differences between 3.6 and 3.7 typing internals. It's also evident to pytypes users that you're very busy on other projects and that this issue is simply a result of your limited time and of the difficulty of the problem, not your ability or commitment to pytypes.

With that in mind, an update as to whether you think there's a chance that this is resolved before python3.6 reaches EOL - and whether there's much any of your many grateful users could do to help by way of development - would be incredibly helpful.

Many thanks again for all your hard work on this indispensable project!

[Stewori]

Hello @aronszanto , thanks for these words. It's good to hear that pytypes is still useful for someone. I am confident that I can fix this issue before Python 3.6 EOL. That will also enable Python 3.8 support. However, there is more to it, i.e. speaking of the general support problem e.g. regarding Python 3.9+. I have no time to write more at the moment, but I will resume on this topic.

[aronszanto]

Hi @Stewori, just a gentle ping here! Looking forward to hearing further thoughts, all best.

[aronszanto]

Hey @Stewori - we're trying to plan for 3.6 EOL. Was wondering if you had any updates as to plans to have pytypes support 3.7/8 by the 3.6 sunset. Thank you!

[Stewori]

3.6 EOL is on December 23, right? I am still confident that I can do a 3.7 and 3.8 compliant release in time (not talking of December 22^^ but I may well become Spetember or October). However at the moment it is not possible for me to spend work here.

[Stewori]

Current master is already mostly 3.7 and 3.8 compatible. In worst case, would it be possible for you to use it as it is right now, maybe packed into a fresh release? There is this single failing test about builtin container types (list, tuple, set, etc) and PEP 484 containertypes without parameters (plain Tuple, List, etc). Maybe this part is not crucial for you...?

[aronszanto]

That might work! Would you consider cutting a mostly-3.7/8-compatible release before following up with the ultimate fix in September/October?

[aronszanto]

Hi @Stewori - any chance you'd release this 3.7/3.8 compatible version as you described? We're coming up on the end of the year :) Thanks!

[Stewori]

I will release current state as "pytypes 1.0b7" probably upcoming week or in the week after November 15th.

[aronszanto]

Thanks so much! Looking forward :)

[Stewori]

Done. Finally turned out as 1.0 beta 8 because I had some trouble with publishing b6 and b7 to PyPI :(

[aronszanto]

Thanks so much! Really appreciate your hard work on this.

[Stewori]

Does someone (@mitar ?)happen to know why the dependecy setuptools_scm is pinned to version 3.5.0? I.e. setuptools_scm==3.5.0? This appears to cause trouble for publishing in conda forge. Github actions still pass if I remove that version-requirement. What consequences might this have? Would it create a less compatible wheel maybe?

[mitar]

I think newer versions do not work on py34. So you will have to drop older Python versions if you want to bump it:

https://github.com/pypa/setuptools_scm/blob/main/CHANGELOG.rst#v400

drop official support for python 3.4

[Stewori]

Okay, thanks for looking that up. I think publishing on conda forge is more important than supporting Python 3.4. So I will file another release with that dependency unpinned.

[Stewori]

Done. Current pytypes is now also available on Anaconda as pytypes 1.0b9.