`pkg_resources` issues with `.deb` package on Ubuntu 22.04

[chabala]

I tried installing from the DEB package, but it seems to have a lot of dependency issues.

I installed via $ sudo apt install ./python3-pcbasic_2.0.5_all.deb

which also installed libsdl2-gfx-1.0-0 python3-parallel python3-serial as prerequisites.

First run:

$ pcbasic
bash: /usr/local/bin/pcbasic: /usr/bin/python: bad interpreter: No such file or directory

Okay, I made a symlink from /usr/bin/python to /usr/bin/python3.

Second run:

$ pcbasic
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
  warnings.warn(
Traceback (most recent call last):
  File "/usr/local/bin/pcbasic", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3267, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3241, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3279, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 573, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 891, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 777, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pcbasic==2.0.5' distribution was not found and is required by the application

I give up here, I'll try installing with pip.

PC-BASIC version: 2.0.5 Operating system version: Ubuntu 22.04 LTS x86_64

[robhagemans]

Not sure what's wrong there, it works out of the box on my system with no complaints.

If Debian packages are system dependent then I'm going to discontinue them in favour of pip as I really can't be checking this on every distribution around.

[chabala]

My guess would be the 'pcbasic==2.0.5' distribution it's complaining about is already satisfied for you if you already have it installed via pip. You can likely easily reproduce the issue by installing the deb in a vanilla ubuntu docker image.

[chabala]

Here is a Dockerfile to reproduce as I mentioned earlier:

# docker build --rm -t pcbasic:2.0.5-deb . && docker run -it --name pcbasic pcbasic:2.0.5-deb
FROM ubuntu:22.04
ADD https://github.com/robhagemans/pcbasic/releases/download/v2.0.5/python3-pcbasic_2.0.5_all.deb ./
ENTRYPOINT ["/bin/python3"]
CMD ["/usr/local/bin/pcbasic"]
RUN set -x \
  && apt-get update -qq \
  && apt install -y --no-install-recommends ./python3-pcbasic_2.0.5_all.deb \
  && rm -r /var/lib/apt/lists /var/cache/apt/archives

This calls python3 directly, as the missing 2.0.5 distribution error is the more interesting issue, but for whatever reason the shebang in /usr/local/bin/pcbasic is set to #!/usr/bin/python, which should be corrected as well. Looks like FPM might need --python-bin python3 to help it pick a better shebang.

[robhagemans]

Thanks for that.

It turns out fpm (which is used to build the .deb) still defaults to Python 2 and needs a few more settings than expected to be convinced to build with Python 3. So the reason the script couldn't find the pcbasic-2.0.5 package is that it has been installed in the site-packages location for Python2.

That also explains why the issue didn't occur on my side as the code is compatible both ways and I still have a Python 2 installed alongside Python 3.

I think I've found the right settings now, it will need some more testing though.

[robhagemans]

Hmm OK, so modules get stored in separate dist-utils per Python version. Meaning that if I build a package on 3.7, and the target system has 3.10, it won't find the package. I'm not sure I can resolve that - I'm not going to distribute separate packages for every live Python version, and the errors you get are pretty opaque so that means lots of future confused users asking questions.

I guess so long as we were using 2.7 this wasn't an issue, as that version stayed the same for a decade. Now new versions come in every year or so and people may be using anything from 3.6 to 3.11...

I'm increasingly thinking the pip wheel is going to be the way forward - the main reason for the packages is that they allow to include a manfile and desktop icons which I think can't/shouldn't be done in a pip wheel, though maybe there's some other way. Ubuntu package maintainers also produce a package on the basis of pip rather than my debs, although of course they don't always have the latest version.

[chabala]

Packaging debs generally and for python specifically are outside my experience. But I'd imagine your concerns are not unique, and they may be upstream issues with fpm, or other fpm users may have ideas about solving them. It's my understanding though that python2 has been discouraged for years now, so I don't think it's safe to assume it will be available. From my perspective as a user, I only have one python version installed, 3.X, whatever version the distro needs and came preinstalled with.

As far as testing that claim, it's a small matter to change the base image the dockerfile uses, so that you could see how your deb performs on ubuntu:18.04 or debian:buster, etc. One could even parameterize the base image and deb, and use it as part of a post-build validation process.

[robhagemans]

I have no intention to assume Python 2 will remain available, or indeed to maintain compatibility for much longer. Its long stability just explains why previously one could release a Python 2.7 package in a deb and it worked.

I can certainly report it upstream, there are a few related open issues in fpm, but it doesn't look like they are moving forward fast.

I think the commonly accepted workflow for debs is actually not to release them at all, but to leave this to Debian/Ubuntu maintainers... That said, I can probably make this work by picking one Python version (say 3.8) and making it an install dependency. For the user it'll be a choice of the Ubuntu deb that's one or two versions behind, the supplied deb that maybe pulls in another Python version, or going through pip.

For background, I don't actually have a big setup for testing packages on various docker instances with different distributions. I barely even get docker running on my somewhat ageing Ubuntu 18.04 install - so the docker setup doesn't really sound like a small matter to me but more like a rather big side project. Really my priority (for this project) is on improving the code itself. Unfortunately the packages have a way of sucking in large amounts time on every release as build tools get upgraded and break and OS assumptions change more rapidly that I can keep up. Fixing packages is an endless cycle of experimenting, as almost all Python packaging tools are poorly documented setuptools hacks.

So what I am really looking for is a way of releasing that just reliably works across systems. This is now a pure-Python package with few dependencies so really it should be possible, but unfortunately this is where Python packaging hell meets Linux desktop packaging purgatory.

[robhagemans]

In a twist of fate, there's not one Python version that's supported across currently live Ubuntu versions (18.04 through 22.04). So no easy way out here.

[robhagemans]

OK, by avoiding fpm altogether and just putting together a shell script using the Debian tools we can create a package with symlinks for all the relevant Python versions. This does appear to work...

[robhagemans]

I expect this now to be solved with release 2.0.6. Please re-open or open a new issue if you still face issue.