lock create doesn't support tar.bz2 extensions

[gruzewski]

After upgrading Pants from 2.15.2 to 2.16.0, I noticed that I cannot generate lock files anymore due to the error:

Unexpected archive type for sdist /tmp/tmp9iayj5vj/tmp.venv.bin.python/python-constraint-1.4.0.tar.bz2

Doing the manual binary search, I found that the last working version of Pex was 2.1.121. Inspecting this change -> https://github.com/pex-tool/pex/pull/2053/files, I see that only .zip and .tar.gz are allowed.

How to reproduce

pex3 lock create python-constraint==1.4.0

I am happy to create a PR to add support for tar.bz2.

[jsirois]

Hrm, well yeah, .tar.bz2 is not supposed to be a thing: https://peps.python.org/pep-0517/#build-sdist

Looking further, even python-constraint realized this eventually and stopped producing wonky sdists.

That said, Pex needs to work with spec violations since the PyPA / spec world is such a mess.

[jsirois]

The most relevant PEP violated here appears to be https://peps.python.org/pep-0625/

[jsirois]

And, the data:

SELECT
REGEXP_EXTRACT(path, r'\.([^.]+|tar\.[^.]+|tar)$') as extension,
COUNT(*) as count
FROM `bigquery-public-data.pypi.distribution_metadata`
GROUP BY extension
ORDER BY count DESC

sdist?	extension	count	% of sdists
whl	6332494
yes	tar.gz	5283102	97.92
egg	135940
yes	zip	108532	2.01
exe	18452
yes	tar.bz2	3857	0.07
msi	625
rpm	603
yes	tgz	226	0.00
dmg	47
deb	36
yes?	tar.zip	2	0.00
yes?	ZIP	1	0.00

[jsirois]

This is noted in #2380, but it appears the chaos here was rooted in distutils / setuptools advice last documented here: https://docs.python.org/3.11/distutils/sourcedist.html

[jsirois]

@gruzewski the fix is now released in Pex 2.2.2. Pants folks should be able to help you upgrade your Pants setup to this Pex release.

[gruzewski]

Awesome, thank you!