Cannot install recently created source distributions.

I have just released Plone 6.0.11. Installing it with Buildout gives an error:

root: Reading https://pypi.org/simple/plone.app.locales/
Getting distribution for 'plone.app.locales==6.0.21'.
While:
  Updating instance.
  Getting distribution for 'plone.app.locales==6.0.21'.
Error: Couldn't find a distribution for 'plone.app.locales==6.0.21'.

When you follow the PyPI link, the last two releases are this:

plone.app.locales-6.0.20.tar.gz
plone_app_locales-6.0.21.tar.gz

Version 6.0.20 from January (included in Plone 6.0.10) can be installed fine. Version 6.0.21, released earlier this week, cannot. The difference is underscores in the filename instead of dots.

A pip install worked fine.

I have fixed it by uploading a wheel, which should have been done anyway, and then it works. The wheel filename is plone.app.locales-6.0.21-py3-none-any.whl, so with dots, which apparently makes it work.

plone.app.locales was released by someone else, but how about packages I released as Plone release manager? Look at plone.base and you see the same. In March I released plone.base-1.3.0.tar.gz, and this week plone_base-1.4.0.tar.gz.

Why is there a difference in the name? Not sure. I use zest.releaser to create releases, usually with a source checkout from the master branch as I am one of the developers of this tool. During release, this basically calls python -m build . This uses the build package. Let's try a bit in a checkout of plone.base. Note that this has a pyproject.toml which since January defines:

[build-system]
requires = ["setuptools>=68.2"]

That may influence things. Let's try to build stuff:

$ python3.12 -mvenv venv
$ . venv/bin/activate
$ pip install build
$ pip freeze --all
build==1.2.1
packaging==24.0
pip==24.0
pyproject_hooks==1.0.0
$ python -m build .
...
Successfully built plone_base-1.4.1.dev0.tar.gz and plone.base-1.4.1.dev0-py3-none-any.whl
$ ls dist
plone.base-1.4.1.dev0-py3-none-any.whl plone_base-1.4.1.dev0.tar.gz

I thought I had seen today with an earlier build version that the source tarball did contain only dots, but I cannot reproduce that, at least not with this package, also not when I remove the build-system from pyproject.toml.

So, I don't know why the filename is suddenly with underscores, but I found PEPs that say this is how it should be:

PEP 625 says that the source distribution filename should be normalised as described in the wheel spec.
The wheel spec says: "In distribution names, any run of -_. characters (HYPHEN-MINUS, LOW LINE and FULL STOP) should be replaced with _ (LOW LINE), and uppercase characters should be replaced with corresponding lowercase ones."

Interestingly the wheel filename does not follow this convention.

But apparently Buildout should be able to install source distribution filenames with underscores, but currently this fails.

To reproduce this, I have created a small namespace package: https://github.com/mauritsvanrees/mauritstest.namespacepackage

I have uploaded version 1.0.0 to PyPI, with only a source distribution: https://pypi.org/simple/mauritstest-namespacepackage/

Wait a minute, the filename there is mauritstest.namespacepackage-1.0.0.tar.gz, so with a dot. That is done by zest.releaser. But when I manually created the source dist, I do get it with an underscore:

$ python3.12 -m venv .
$ . venv/bin/activate
$ python -m build .
...
Successfully built mauritstest_namespacepackage-1.0.0.dev0.tar.gz and mauritstest.namespacepackage-1.0.0.dev0-py3-none-any.whl
$ ls dist/
mauritstest.namespacepackage-1.0.0.dev0-py3-none-any.whl mauritstest_namespacepackage-1.0.0.dev0.tar.gz

So yet another mystery: why does zest.releaser create an sdist with an underscore, and the normal build not? An why do my recent Plone releases, made with zest.releaser, have underscores, and this one not?

Let's not get side tracked by that. I create a source dist manually, with underscore, and upload it with twine. Done with version 1.0.1.

So now this buildout.cfg works:

[buildout]
parts = test

[test]
recipe = zc.recipe.egg
eggs = mauritstest.namespacepackage

[versions]
mauritstest.namespacepackage = 1.0.0

But when I edit it to use 1.0.1 it fails:

$ bin/buildout 
Updating test.
Getting distribution for 'mauritstest.namespacepackage==1.0.1'.
While:
  Updating test.
  Getting distribution for 'mauritstest.namespacepackage==1.0.1'.
Error: Couldn't find a distribution for 'mauritstest.namespacepackage==1.0.1'.

So now it should be reproducible.

For completeness sake, this buildout uses Python 3.11.8 on Mac OS 14.4.1, and these package versions in the venv:

$ bin/pip freeze --all
pip==24.0
setuptools==65.5.0
wheel==0.43.0
zc.buildout==3.0.1

Interesting difference:

1. Executing the build module:

$ python -m build --sdist .
...
Successfully built mauritstest_namespacepackage-1.0.1.tar.gz

So underscore.

2. Using ProjectBuilder:

This is how zest.releaser does it, calling build.ProjectBuilder:

$ python
>>> from build import ProjectBuilder
>>> builder = ProjectBuilder(source_dir=".")
>>> builder.build("sdist", "./dist/")
...
'/Users/maurits/own/mauritstest.namespacepackage/dist/mauritstest.namespacepackage-1.0.1.tar.gz'

So with dots.

Difference

Aha, a difference is that python -m build creates an isolated environment by default. And using ProjectBuilder, you use the current Python environment. That current env has setuptools 65.7.0.

We can run build without isolation:

$ python -m build --sdist --no-isolation .
...
Successfully built mauritstest.namespacepackage-1.0.1.tar.gz

We have a dot here now. So the version of setuptools matters. Without isolation, you presumably get the latest, which now is 69.5.1. With that, both methods produce underscores.

setuptools 69.2.0 is the last release that still produces dots. And in the Plone core development buildout, I updated setuptools to a higher version last week. That explains the difference.

The setuptools history says for version 69.3.0: "Support PEP 625 by canonicalizing package name and version in filenames. (#3593)"

That issue has an informative comment about project names to give some background on why it is tricky for setuptools to do the right thing, and there are always compromises.

69.3.0 was actually yanked from PyPI, but that was for a different reason, to do with normalisation of version numbers. Latest 69.5.1 has the (for us) problematic code.

Note that the setuptools version only matters at the moment that the source dist is generated. It does not matter which version you use in the Buildout where you try to install the source dist.

Note that wheel names still have dots, but that may change.

More reading: https://github.com/pypa/setuptools/issues/3777

Anyway, after all that, the point is: Buildout should be able to install source distributions (and wheels) that have underscores instead of dots in the filename.

buildout / buildout