Install broken due to python changes in Raspbian Bookworm

[JonCellini]

Describe the problem Due to changes in how python functions in Raspbian Bookworm the default install script fails to install the python dependencies due to a change in how that is handled by default.

It gives the following error:

error: externally-managed-environment

The script continues but SlowMovie will fail to run as it's dependencies are not properly met.

Cloning into '/home/pi/SlowMovie'...
remote: Enumerating objects: 1479, done.
remote: Counting objects: 100% (543/543), done.
remote: Compressing objects: 100% (168/168), done.
remote: Total 1479 (delta 474), reused 375 (delta 375), pack-reused 936
Receiving objects: 100% (1479/1479), 5.55 MiB | 7.96 MiB/s, done.
Resolving deltas: 100% (832/832), done.
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
SlowMovie install/update complete. To test, run 'python3 /home/pi/SlowMovie/slowmovie.py'
Created symlink /etc/systemd/system/multi-user.target.wants/slowmovie.service → /etc/systemd/system/slowmovie.service.
SlowMovie service installed! Use sudo systemctl start slowmovie to test 

Steps to Reproduce Install a fresh Raspbian Bookworm image. Run the installer script.

Platform Information Raspberry Pi 3 model A+ Raspbian Bookworm 2023-10-10 (reproducible on both armhf & arm64) Python 3.11.2

[robweber]

Doing a bit of research on this it seems this is a direct result of PEP 668. A good (and short) explanation of it is here.

The current install script, and manual instructions, have users installing Python packages within the user space. This is discouraged over the user of virtual environments. There are some hacky ways of just removing the EXTERNALLY-MANGED file to avoid this error but modifying things to create and use a virtual environment is the proper fix.

[MagnusOstertag]

Using an python environment I get the following error:

  Running setup.py install for IT8951 ... error
  error: subprocess-exited-with-error

  × Running setup.py install for IT8951 did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running install
      /home/pi/SlowMovie/venv/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-armv7l-cpython-311
      creating build/lib.linux-armv7l-cpython-311/IT8951
      copying IT8951/constants.py -> build/lib.linux-armv7l-cpython-311/IT8951
      copying IT8951/interface.py -> build/lib.linux-armv7l-cpython-311/IT8951
      copying IT8951/__init__.py -> build/lib.linux-armv7l-cpython-311/IT8951
      copying IT8951/display.py -> build/lib.linux-armv7l-cpython-311/IT8951
      running build_ext
      building 'IT8951.spi' extension
      creating build/temp.linux-armv7l-cpython-311
      creating build/temp.linux-armv7l-cpython-311/IT8951
      arm-linux-gnueabihf-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/pi/SlowMovie/venv/include -I/usr/include/python3.11 -c IT8951/spi.c -o build/temp.linux-armv7l-cpython-311/IT8951/spi.o
      IT8951/spi.c: In function ‘__Pyx_TraceSetupAndCall’:
      IT8951/spi.c:22082:37: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
      22082 |         if (CYTHON_TRACE && (*frame)->f_trace == NULL) {
            |                                     ^~
      IT8951/spi.c:22084:21: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
      22084 |             (*frame)->f_trace = Py_None;
            |                     ^~
      IT8951/spi.c:438:62: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
        438 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
            |                                                              ^~
      IT8951/spi.c:22091:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
      22091 |     __Pyx_PyFrame_SetLineNumber(*frame, firstlineno);
            |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      IT8951/spi.c: In function ‘__Pyx_PyBytes_Equals’:
      IT8951/spi.c:23312:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
      23312 |             hash1 = ((PyBytesObject*)s1)->ob_shash;
            |             ^~~~~
      In file included from /usr/include/python3.11/bytesobject.h:62,
                       from /usr/include/python3.11/Python.h:50,
                       from IT8951/spi.c:6:
      /usr/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
          7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
            |                                   ^~~~~~~~
      IT8951/spi.c:23313:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
      23313 |             hash2 = ((PyBytesObject*)s2)->ob_shash;
            |             ^~~~~
      /usr/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
          7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
            |                                   ^~~~~~~~
      IT8951/spi.c: In function ‘__Pyx_AddTraceback’:
      IT8951/spi.c:438:62: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
        438 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
            |                                                              ^~
      IT8951/spi.c:24384:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
      24384 |     __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
            |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      error: command '/usr/bin/arm-linux-gnueabihf-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> IT8951

[robweber]

That error is specifically with the IT8951 EPD drivers. I imagine something in their build process has changed with the Python version on Bookworm. I haven't had time to play with this much myself but do intend on testing it out and making modifications.

[wobblewoo]

Im not sure its advised, but after running into this issue, i have found a solution.

Add this to the end of the pip commands: --break-system-packages

pip3 install "git+https://github.com/waveshare/e-Paper.git#subdirectory=RaspberryPi_JetsonNano/python&egg=waveshare-epd" --break-system-packages

Source: https://github.com/python/cpython/issues/102134

Here be dragons...read this part at your own risk!
If you're really, really, really sure you want to use pip or another non-system package manager directly on the packages installed by your Linux distribution, you can pass the --break-system-packages flag to pip. However, it will do exactly what it says if you aren't extremely careful, so if you aren't 120% sure you're an expert who knows exactly what you're doing, don't use it. And if you do, don't say I didn't warn you! :)

[wobblewoo]

After a fresh install on a new Pi (using --break-system-packages on the pip commands) i now get this error, Can anyone help please? raspberry@raspberrypi:~/SlowMovie $ python3 slowmovie.py Traceback (most recent call last): File "/home/raspberry/SlowMovie/slowmovie.py", line 24, in <module> from omni_epd import displayfactory, EPDNotFoundError ModuleNotFoundError: No module named 'omni_epd

[robweber]

Seems like the libraries most likely didn't installed properly. I wouldn't advise using the --break-system-packages command and it's not likely to be the long term solution.

Raspberry Pi OS - Bookworm has only been out for a few weeks so until we've had a chance to update the install instructions please be patient. It's likely some downstream libraries aren't yet supported. You can still get Bullseye from the archive site as that is known to work.

[wobblewoo]

That was it, Bullseye worked. Didn't even notice the new version. Dog!

Thankyou

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Rob @.> Sent: Sunday, October 29, 2023 2:45:07 PM To: TomWhitwell/SlowMovie @.> Cc: Warren @.>; Comment @.> Subject: Re: [TomWhitwell/SlowMovie] Install broken due to python changes in Raspbian Bookworm (Issue #150)

Seems like the libraries most likely didn't installed properly. I wouldn't advise using the --break-system-packages command and it's not likely to be the long term solution.

Raspberry Pi OS - Bookworm has only been out for a few weeks so until we've had a chance to update the install instructions please be patient. It's likely some downstream libraries aren't yet supported. You can still get Bullseye from the archive sitehttps://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2023-05-03/ as that is known to work.

— Reply to this email directly, view it on GitHubhttps://github.com/TomWhitwell/SlowMovie/issues/150#issuecomment-1784133550, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AT7CWHTN4WE7MH2HVKNDXXLYBZTXHAVCNFSM6AAAAAA54YR5DCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUGEZTGNJVGA. You are receiving this because you commented.Message ID: @.***>

[JonCellini]

The current install script, and manual instructions, have users installing Python packages within the user space. This is discouraged over the user of virtual environments.

Hey Rob,

I agree a venv is the way to go. That doesn't seem like too heavy of a lift for my shell scripting skills, would you guys be open to a contribution of an enhancement to the install script logic to handle this?

[JonCellini]

× Encountered error while trying to install package. ╰─> IT8951

I ran into that issue on Bookworm as well and was able to resolve it a moment ago while working on the PR for moving to a python venv. The solution is mentioned in the upstream repo here. I'll write up a proper PR for fixing that as well and submit it :)

[robweber]

Awesome. The omni-epd library is the upstream where that import is happening. Nothing is ever easy is it? https://github.com/robweber/omni-epd

Do you think this will make trouble for < Python 3.11 installs? Might have to raise the min Python version if it won't work downstream.

[JonCellini]

I'm testing it now on a real Pi and will submit a PR for that as soon as it's verified to be working reliably as part of the install (I'd fixed my local copy by hand). My python is somewhat shallow but with a virtual environment you can pin the Python version so I don't imagine it should be a problem. I had success with Python 3.11.2 on my local Bookworm install.

[JonCellini]

While the change will fix the install of the IT8951 on Bookworm it throws an error when trying to run it - I was focused on testing the venv change so I didn't actually try to start the service post-install. I'll poke around at it more as time permits this week.

[robweber]

@JonCellini's PR handles all the virtual environment pieces; however the waveshare library currently has a number of issues related to Bookworm that require holding off on merging anything in a good idea. Links below but it appears that on install, and in several key pieces of code, the Waveshare library attempts to check if the hardware is an Rpi or a Jetson Nano. These checks fail on Bookworm as they're checking for OS files that just don't exist anymore. As a result Nano dependencies and code are attempting to be loaded instead of Rpi stuff. This is all within the library itself and not something that we can work around. For now just have to watch the relevant Waveshare PRs.

https://github.com/waveshareteam/e-Paper/issues/306 https://github.com/waveshareteam/e-Paper/pull/307

[GregDMeyer]

Hi all, Just wanted to call attention to the new commit I just pushed it IT8951---hopefully it will resolve some of the issues you were having!

[robweber]

Great - thanks @GregDMeyer!

Once the Waveshare stuff is ironed out to work on Bookworm too we should be able to move #151 along and get this project going on those systems again.

[robweber]

I've made some modifications to the omni-epd package that I've tested on both Bullseye and Bookworm. Everything seems to build and work on both those systems on my end. I'm testing using a Waveshare 7.5in display, I don't have an IT8951 type display but the library builds so guessing they'll work properly.

If anyone wants to test this you can use this branch and install into a virtual environment in Bookworm. I changed the Waveshare library from the official one to a fork containing fixes for Bookworm specifically. These fixes work on Bullseye as well so nothing should break on existing Bullseye installs. Not the ideal fix but when the official Waveshare repo gets updated it will be a quick swap out to get the original source working again.

If you're testing this on an existing install the easiest thing to do is create a new virtual environment and then change the omni-epd line in the requirements.txt file to reflect the new branch and then install all the requirements normally and run.

# change source to this in requirements.txt
git+https://github.com/robweber/omni-epd.git@bookworm_updates#egg=omni-epd

# build
pip3 install -r Install/requirements.txt