Switch to PEP 420 (Implicit namespace packages)

[icemac]

Currently we use pkg_resources-style namespace packages. Using pkg_resources is deprecated (see https://setuptools.pypa.io/en/latest/pkg_resources.html), it seems not yet clear when it is removed from setuptools.

The current future are PEP 420: Implicit namespace packages.

This requires:

• all the packages sharing a namespace to switch over to PEP 420.
• removing the __init__.py files existing to create a namespace
• removing namespace_packages declaration from setup.py
• maybe removal of the dependency on setuptools? (Maybe we could even switch to pyproject.toml.)

zopefoundation hosts the following namespaces:

• [ ] zope, zope.app – 117 repos
• [ ] z3c, z3c.recipe, z3c.layer, z3c.formwidget, z3c.menu – 43 repos
• [ ] zc, zc.recipe – 22 repos
• [ ] Products – 22 repos
• [ ] keas (probably only some packages of that namespace) – 1 repo
• [ ] refline (probably only some packages of that namespace) - 1 repo
• [ ] grokcore – 17 repos
• [X] grokui – all repos archived
• [ ] hurry – 2 repos
• [ ] cipher – 2 repos
• [ ] five – 4 repos
• [X] megrok: 1 repos (https://github.com/zopefoundation/megrok.strictrequire/pull/8)

See the discussion in https://github.com/pypa/setuptools/pull/3434 for some more details.

Suggestion:

• Try out on a small namespace where probably all packages are hosted by zopefoundation (grokcore?, z3c?, zope.app?)
• If this works fine (especially in zc.buildout), adapt the other namespaces.
  • This will require coordination with the Plone folks.
  • This would also require a big warning sign that other packages hosted outside zopefoundation but using the same namespaces also have to switch to PEP 420.

Cc-ing: @dataflake @mauritsvanrees @gforcada @mgedmin @jensens

See also https://packaging.python.org/en/latest/guides/packaging-namespace-packages/

[dataflake]

This looks like a ton of work... but if we could really get rid of setuptools it would be worth it. Setuptools is nothing but trouble at this point.

[mauritsvanrees]

I left two comments in the setuptools discussion, with some numbers. Let's see if we can get a timeline for when setuptools will drop support.

Zope and Plone can only move to native namespaces in a major release. Since Plone 6 was only released a few months ago, I expect it will take two to three years before we have Plone 7, which would need a Zope 6. We may need to speed that up. But I hope we can let this rest for another year.

If we know that for example setuptools 70 is going to drop support, I suppose we could add setuptools<70 in install_requires to prevent installation of incompatible packages side by side.

[jensens]

+1 to start the process of modernization here. Some remarks:

• I agree, this change is necessary at some point.
• I agree with @mauritsvanrees that this is a breaking change at some point and would need a new major Zope release.
• I think a good idea is to introduce this change incrementally:
  • first all controlled zope.* packages can be changed relatively easy, probably the same is valid for five.*, z3c.* and zc.* (and maybe some other in the list above I do not know/use). This might go into the Zope 5 release series.
  • With the Products.* namespace it gets difficult, this is a bunch of packages, including CMF, Plone and all add-ons. It needs tight coordination.
  • If we switch everything to PEP420 in Plone this would be possible only in a Plone 7.
• if this topic is touched I would switch to full to pyproject.yml w/o any setup.* along with it. For my company projects I use already PEP420 style packages along with pyproject.toml w/o any problems.
• ad getting rid of Setuptools: What is the idea behind this?
  • Setuptools is settled, widely used, has lots of tooling available and is relatively fast.
  • What is the alternative we want? We need some PEP 518 [build-system]. I am not that a fan of Poetry, did not try Flit or PDM.

[dataflake]

The idea of getting rid of setuptools was not based on any knowledge of replacements, but on some comments in that setuptools discussion on GitHub. It may not be possible. Where I come from is endless issues with the Zope packages that use C-code in conjunction with macOS and the different builds that exist for it (x86_64, aarch64 and universal2). I described that at https://github.com/zopefoundation/meta/issues/181.

[icemac]

I am not aware of an alternative to setuptools which can handle C code. But I am open to learn if there is one.

[mgedmin]

Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?

Are we talking about a flag day here, with major version bumps and coordinated releases for every zope.* package at the same time? (And similar efforts for zc.*, z3c.*, grok.*, collective.* namespaces)?

[d-maurer]

Marius Gedminas wrote at 2023-3-8 23:31 -0800:

Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?

As I read PEP-420 it does not provide a transistion path with pkg_resources but it does for pkgutil.extend_path -- with the limitation that in this case, __path__ recomputation is not guaranteed (maybe: will not happen).

It might be possible that pkg_resources.declare_namespace has been extended in modern setuptools versions to handle implicit namespace packages correctly. However, I am not sure: implicit namespace packages make __path__ read-only which may make the task for declare_namespace much more difficult.

[d-maurer]

Marius Gedminas wrote at 2023-3-8 23:31 -0800:

Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?

I have had a second thought. When I understand PEP-420 correctly, then it modifies the import of package p as follows:

• it looks for p on the current __path__
• if it finds a directory p with __init__, then p is instantiated as a "normal" package
• only if all directories for p lack a __init__, it is instantiated as a namespace package.

This would imply: either all directories for a namespace package p lack an __init__ or the existing __init__s must emulate the namespace package treatment.

Accoding to PEP-420, pkgutil.extend_path can be expected to partially emulate the namespace package treatment (with the exception of support for subsequent __path__ changes). Modern setuptools versions might have a pkg_resources.declare_namespace which does the same.

[jensens]

Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?

I did not read the PEPs in detail nor the code about it, but about a year ago I just tried it and it just did not work. Ago then, I found some source-on-the-internet stating exact: it does not mix (but currently I can not find it again).

[mauritsvanrees]

That would be the Python Packaging User Guide, which says:

Warning: While native namespace packages and pkgutil-style namespace packages are largely compatible, pkg_resources-style namespace packages are not compatible with the other methods. It’s inadvisable to use different methods in different distributions that provide packages to the same namespace.

It also gives one specific reason for using pkg_resources-style namespace packages: "This method is recommended if you need compatibility with packages already using this method or if your package needs to be zip-safe."

Do we need that for some reason? I guess not: in a lot of Zope and Plone package I see zip_safe=False, so the other way around.

[jensens]

"This method is recommended if you need compatibility with packages already using this method or if your package needs to be zip-safe."

Do we need that for some reason?

"It is very unlikely that the values of zip_safe will affect modern deployments that use pip for installing packages. Moreover, new users of setuptools should not attempt to create egg files using the deprecated build_egg command. Therefore, this flag is considered obsolete." https://setuptools.pypa.io/en/latest/deprecated/zip_safe.html

[graingert]

for the jaraco. namespace @jaraco was able to do a staged migration to pkgutil namespace packages see https://github.com/pypa/setuptools/pull/3434#issuecomment-1435697632

I think the packaging tutorial docs may be incorrect as there's a test for pkg_resources and pkgutil namespaces in one virtual environment https://github.com/pypa/sample-namespace-packages/blob/df7530eeb8fa0cb7dbb8ecb28363e8e36bfa2f45/noxfile.py#L79-L88 which shows a staged migration is possible

[graingert]

There's now work to test staged migration of namespace packages in https://github.com/pypa/sample-namespace-packages/pull/22 referring to the table it looks like as long as you don't mind about not supporting pip install -e . a staged migration to pep 420 is supported

[mauritsvanrees]

Let's see if I understand it correctly.

So "staged migration" means: migrate some packages in a namespace to pep420 while some others still use pkg_resources?

For Zope/Plone in that table the lines near the bottom with "cross_pkg_resources_pep420" are the relevant ones. I don't see a difference in behavior for Python versions, so that is good. And you can use either a normal or editable install for old pkg_resources packages, but the pep420 ones then only work with normal installs. So editable installs can be tricky, but a standard pip install Zope or Plone will just work, regardless of which type of namespaces are used.

That is better than I had hoped for!

It does make it a bit tricky in the transition period. Say you want to start with the Products namespace and start working on the first package. You do an editable install of the old code, and it works. Then you edit setup.py to move to native namespaces. Now you can no longer do an editable install. That makes it harder to check that it actually works before making a release. Well, pip install . should still work, so it is not the end of the world. But when you want to fix an actual bug in the code, then after editing your files you need to explicitly run pip install . again, which is a pain. So we should still try to keep the transition period short.

[icemac]

Heads up: zope.testrunner currently does not support implicit namespaces, see https://github.com/zopefoundation/zope.testrunner/issues/160

[jensens]

Btw., I wrote I small blog about the mix of old and new style packages in one namespace problem last year, while confronted with this problem while working on a different project: https://yenzenz.com/blog/2023/mixed_python_pep420/

[malthe]

Here's a vote for pip install -e. – I use that all the time!

[icemac]

I updated the list of needed packages to be ported in the description of the issue.

[npilon]

I've gotten a lot of good use out of various zope packages, especially zope.sqlalchemy and zope.interfaces, so I'd love to help with this migration. It looks like buildout might also need some adjustment - or replacement? How can a new volunteer chip in?

[dataflake]

Thanks for the offer. It looks like right now there's still too much confusion and no actual plan.

[npilon]

Thanks for the offer. It looks like right now there's still too much confusion and no actual plan.

Is there anywhere central discussion or planning is happening? From trying to PEP 420 zope.sqlalchemy a bit yesterday it feels like adjusting zc.buildout to work with current packaging conventions would maybe be the place to start?

[d-maurer]

Nicholas Pilon wrote at 2024-6-8 05:20 -0700:

... Is there anywhere central discussion or planning is happening? From trying to PEP 420 zope.sqlalchemy a bit yesterday it feels like adjusting zc.buildout to work with current packaging conventions would maybe be the place to start?

I think making zc.buildout cope with modern packaging practices is a good idea (even though I no longer use zc.buikdout), but it is orthogonal to (independent of) implicit namespace packages.

Implicit namespace packages by itself is not a big problem. The problem is that all subpackages of a namespace package must switch at the same time to implicit namespaces (in any given installation). Thus, if you have e.g. zope.event with explicit namespaces installed and then you upgrade zope.interface and get for it an implicit namespace, strange things can happen.

[npilon]

The blog post from https://github.com/zopefoundation/meta/issues/194#issuecomment-1914638141 just describes problems with editable installs and https://github.com/zopefoundation/meta/issues/194#issuecomment-1470346176 seems to indicate it should work - again, outside of mixed editable installs.

Are there other documented strange behaviors? If it’s just mixed editable installs it seems like the right approach is just upgrade and maybe have minimum versions in requirements on cross dependencies?

[d-maurer]

Nicholas Pilon wrote at 2024-6-8 09:18 -0700:

... Are there other documented strange behaviors? If it’s just mixed editable installs it seems like the right approach is just upgrade and maybe have minimum versions in requirements on cross dependencies?

I have not yet worked with implicit namespace packages. But I have seen serious problems with (explicit) namespace packages when some subpackages have been installed via pip and others via zc.buildout: strange import errors occured. I failed to understand the problem details but I assume that the path extension logic cannot cope with this situation.

The implicit namespace PEP speaks of a fullback for subpackages not yet using implicit namespaces BUT warns against its use: Normally, the __path__ of an implicit namespace package is automatically updated when the parent path changes. However, the fallback changes __path__ itself and this disables future automatic updates of it. I fear that thereafter adding additional implicit namespace subpackages to the parant path no longer updates __path__ and import errors are to be expected.

[npilon]

That’s actually exactly what I ran into trying to tinker together a simple solution for one zope subpackage yesterday so I think I have some idea about what’s going on; I’m hoping that starting with zc.buildout I can bypass that.

[d-maurer]

Nicholas Pilon wrote at 2024-6-8 11:45 -0700:

That’s actually exactly what I ran into trying to tinker together a simple solution for one zope subpackage yesterday so I think I have some idea about what’s going on; I’m hoping that starting with zc.buildout I can bypass that.

I am less optimistic that zc.buildout alone can fully support mixed namespace packages -- unless it transforms explicit namespace packages to implicit ones on installation.

We will still have the problem of packages partly installed via zc.buildout and partly externally (via the OS, via pip) -- but this problem already exists in the "all explicit" world.