search

jaraco / jaraco.functools

MIT License
17 stars 14 forks source link

Convert all projects to `pkgutil`-based namespace #9

Closed jaraco closed 5 years ago

jaraco commented 5 years ago

This ticket applies to this project as well as all other projects in the jaraco namespace. I'd like to migrate these projects to use the pkgutil-style technique for the jaraco namespace.

I've done some tests and it does seem that the two techniques are in fact compatible enough when pip installed. I did two tests:

  1. Ran the tests of jaraco.functools using pkgutil namespaces with jaraco.classes (a dependency) still using pkg_resources namespaces.
  2. Ran the tests of jaraco.functools still using pkg_resources namespaces, but with an unreleased version of jaraco.classes using pkgutil namespaces.

In both cases, on both Python 3.7 and 2.7, the tests passed without issues.

This test leads me to believe that in the general case, these namespace techniques are compatible. I know there are compatibility issues when invoking setup.py install, but that technique is deprecated. Based on this finding, I'd like to proceed with making the shift in all of the packages.

I do plan to still make a backward-incompatible bump for this transition so it's possible for packages to pin to older versions with the pkg_resources technique or select into the newer pkgutil technique.

jaraco commented 5 years ago

I've completed the conversion on most of them. The following remain:

jaraco.compat jaraco.crypto jaraco.desktop jaraco.develop jaraco.docker jaraco.email jaraco.fabric jaraco.geo jaraco.home jaraco.imaging jaraco.input jaraco.institutions jaraco.media jaraco.net jaraco.nxt jaraco.office jaraco.path jaraco.util jaraco.video

jaraco commented 5 years ago

It's worth writing up why I wanted to make this change.

The main reason is that the pkg_resources-style technique is deprecated... and that technique has issues, such as requiring import of pkg_resources, which has been traditionally an implicit dependency with import-time performance challenges.

I'm working to remove reliance on pkg_resources, and this is one area where the best recommendation for removing that reliance is to switch the technique used.

I've also been perpetually reminded by pytest-dev/pytest#4014 that these implicit startup-time imports were creating spurious warnings in tests, warnings for which the best recommendation is not to use pkg_resources-style namespaces.

And of course, as one of the contributors to the PEP 420 namespace packages, I'd love to take advantage of those (once Python 3.5 can be relied upon), but that's not possible while other packages in the jaraco namespace are using the pkg_resources technique.

Mic92 commented 5 years ago

Mhm. That seems to break for NixOS:

$ echo $PYTHONPATH
/nix/store/hqgbdad34lm9gpxz0726bf7rw4yzp9gm-python3.7-irc-17.0/lib/python3.7/site-packages:/nix/store/5501pjsrvsbygsy019hclc4j85q0dljb-python3.7-six-1.11.0/lib/python3.7/site-packages:/nix/store/yv4pzx3lxk3lscq0pw3hqzs7k4x76xsm-python3-3.7.2/lib/python3.7/site-packages:/nix/store/i1cqs1flnain593sk6m5vh8jigxpmqcx-python3.7-setuptools-40.6.3/lib/python3.7/site-packages:/nix/store/ilfkd29ygbxbsp1zg1phv6qrpp9jskvq-python3.7-jaraco.logging-2.0/lib/python3.7/site-packages:/nix/store/g7q02x0hw6i16x36swz65ia4an109bs3-python3.7-tempora-1.14/lib/python3.7/site-packages:/nix/store/wcspmzkyl94k0cbi3lqf81gap95hq4gr-python3.7-pytz-2018.7/lib/python3.7/site-packages:/nix/store/lw4w4hic1m1h47an95z24cq2r3rmjipv-python3.7-jaraco.functools-1.20/lib/python3.7/site-packages:/nix/store/rq9g7al68dam0g1r15mnb7n1zc19xfkw-python3.7-more-itertools-4.3.0/lib/python3.7/site-packages:/nix/store/5g7zxwwfzax6yji2hviflvnv5pwibm7l-python3.7-jaraco.text-1.10.1/lib/python3.7/site-packages:/nix/store/zm79il4r57x3p9fvdig9vxawn7bf0yhf-python3.7-jaraco.collections-1.6.0/lib/python3.7/site-packages:/nix/store/xc8cy5yfylxpafbgbspc02i866xz8bx4-python3.7-jaraco.classes-2.0/lib/python3.7/site-packages:/nix/store/3sarmwvz5mqsyb8ibp62jf4m9myzh952-python3.7-jaraco.stream-1.2/lib/python3.7/site-packages:/nix/store/dl8c2z46rlljwiw5g58ghkyqwp7w5sf8-python3.7-jaraco.itertools-4.0.0/lib/python3.7/site-packages:/nix/store/7frml604gl2lbbp36n4n8iawfa0ch80v-python3.7-inflect-1.0.2/lib/python3.7/site-packages:/nix/store/hqgbdad34lm9gpxz0726bf7rw4yzp9gm-python3.7-irc-17.0/lib/python3.7/site-packages:/nix/store/5501pjsrvsbygsy019hclc4j85q0dljb-python3.7-six-1.11.0/lib/python3.7/site-packages:/nix/store/yv4pzx3lxk3lscq0pw3hqzs7k4x76xsm-python3-3.7.2/lib/python3.7/site-packages:/nix/store/i1cqs1flnain593sk6m5vh8jigxpmqcx-python3.7-setuptools-40.6.3/lib/python3.7/site-packages:/nix/store/ilfkd29ygbxbsp1zg1phv6qrpp9jskvq-python3.7-jaraco.logging-2.0/lib/python3.7/site-packages:/nix/store/g7q02x0hw6i16x36swz65ia4an109bs3-python3.7-tempora-1.14/lib/python3.7/site-packages:/nix/store/wcspmzkyl94k0cbi3lqf81gap95hq4gr-python3.7-pytz-2018.7/lib/python3.7/site-packages:/nix/store/lw4w4hic1m1h47an95z24cq2r3rmjipv-python3.7-jaraco.functools-1.20/lib/python3.7/site-packages:/nix/store/rq9g7al68dam0g1r15mnb7n1zc19xfkw-python3.7-more-itertools-4.3.0/lib/python3.7/site-packages:/nix/store/5g7zxwwfzax6yji2hviflvnv5pwibm7l-python3.7-jaraco.text-1.10.1/lib/python3.7/site-packages:/nix/store/zm79il4r57x3p9fvdig9vxawn7bf0yhf-python3.7-jaraco.collections-1.6.0/lib/python3.7/site-packages:/nix/store/xc8cy5yfylxpafbgbspc02i866xz8bx4-python3.7-jaraco.classes-2.0/lib/python3.7/site-packages:/nix/store/3sarmwvz5mqsyb8ibp62jf4m9myzh952-python3.7-jaraco.stream-1.2/lib/python3.7/site-packages:/nix/store/dl8c2z46rlljwiw5g58ghkyqwp7w5sf8-python3.7-jaraco.itertools-4.0.0/lib/python3.7/site-packages:/nix/store/7frml604gl2lbbp36n4n8iawfa0ch80v-python3.7-inflect-1.0.2/lib/python3.7/site-packages
$ cat /nix/store/dl8c2z46rlljwiw5g58ghkyqwp7w5sf8-python3.7-jaraco.itertools-4.0.0/lib/python3.7/site-packages/jaraco/__init__.py
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
$ ls -la /nix/store/dl8c2z46rlljwiw5g58ghkyqwp7w5sf8-python3.7-jaraco.itertools-4.0.0/lib/python3.7/site-packages/jaraco/
total 43
dr-xr-xr-x 3 root root     5 Jan  1  1970 .
dr-xr-xr-x 4 root root     4 Jan  1  1970 ..
dr-xr-xr-x 2 root root     4 Jan  1  1970 __pycache__
-r--r--r-- 1 root root    65 Jan  1  1970 __init__.py
-r--r--r-- 1 root root 22131 Jan  1  1970 itertools.py
$ python3 -c 'from jaraco.itertools import always_iterable, infinite_call' 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'jaraco.itertools'

We have each package in its own directory and our tooling will then collect it from their to build the final PYTHONPATH. cc @FRidh might have more insights on this.

jaraco commented 5 years ago

I suspect the issue is that one or more of those jaraco.* dependencies are stuck at an older version that doesn’t yet have the pkgutil technique. I’ll bet if you import jaraco and then inspect jaraco.__path__, it will reveal which packages are still using the pkg_resources techique and need to be upgraded.

FRidh commented 5 years ago

We've indeed encountered this before, and will have to update packages and optionally patch them to use the same technique.

jaraco commented 5 years ago

Sorry for the inconvenience. I did do some things that I hope will ease the transition:

I realize from a systems point of view, this is a bit of a mess, but my hope was to make it clean (for the reasons above). Let me know if there's more I can do to help.

FRidh commented 5 years ago

Thanks for the explanation @jaraco. We did notice one other issue, and that's a circular runtime dependency between jaraco_collections and jaraco_text. Was this intentional? We can build wheels separately for them (as we currently do), however, it requires us to install them together, which is a bit unfortunate, as we typically have a separate site-packages folder for each package (cc @dotlambda).

jaraco commented 5 years ago

I've moved the discussion on circular dependencies to a relevant project ^.