mopidy/install.sh: broken on Bookworm

Gadgetoid commented 9 months ago

I don't know what the workaround might be, but the changes to Bookworm that prevent pip from installing packages system-wide mean it's impossible to install Mopidy plugins from pypi.

That means the install script can't add essential plugins for Pirate Audio: Mopidy-PiDi Mopidy-Local pidi-display-pil pidi-display-st7789 mopidy-raspberry-gpio, which happens here: https://github.com/pimoroni/pirate-audio/blob/d9d7cadf1686860dcbfe42b74d34454942cba2d4/mopidy/install.sh#L126C14-L126C14

Right now I don't know what the path to fixing this is, and I'm still working out how we approach virtual environments across our products.

So -

Pirate Audio + Mopidy will currently not work on Pi 5, for this and other reasons
Pirate Audio + Modpify will not work on Bookworm OS in a Pi 4, though just audio output is possible

In the mean time I recommend using Raspberry Pi OS Bullseye, which you can still get from here:

http://downloads.raspberrypi.org/raspios_oldstable_full_armhf/images/raspios_oldstable_full_armhf-2023-10-10/

L0stCh1ld commented 7 months ago

there is a dirty kludge fix for now on bookworm. edit 'install.sh' and locate the lines beginning with ' $PIP_BIN install --upgrade ' edit them like so ' $PIP_BIN install --break-system-packages --upgrade '

amigauser1 commented 5 months ago

Another way is to add the following line at the beginning (after #!/bin/bash) of install.sh:

export PIP_BREAK_SYSTEM_PACKAGES=1

This avoids adding the break-system-option to all pip lines.

kingosticks commented 4 months ago

@Gadgetoid I'd like to fix this (including playback), am I OK to go with either of the last two options? Or something else?

theNizo commented 3 months ago

Tried venvs - mopidy can't find gstreamer
Tried installing everything via pip - doesn't change the fact that the EXTERNALLY-MANAGED already exists in the image you flash. You have to install gstreamer1.0-alsa for it to work additionally, and you don't get a systemd-service automatically
Tried with the flag - seems to be working best. GPIO project doesn't work on the newest version of python because of course it doesn't, so buttons don't work.

kingosticks commented 3 months ago

Tried venvs - mopidy can't find gstreamer

You can if you create the venv with --system-site-packages

Tried installing everything via pip

I see you've deleted your original post about this, but yes, this doesn't help. Externally-managed is about not installing non-apt packages into the system Python space, sudo pip does nothing to avoid this.

Ignoring externally-managed is the ~~only~~ easiest solution. There are multiple way to ignore it. The non-permanent ways are best, in my opinion. But until there's some appetite for accepting these changes, I'm not going to spend time on it.

theNizo commented 3 months ago

You can if you create the venv with --system-site-packages

Oh, thank you, maybe I'll try that.

I see you've deleted your original post

don't want to post incomplete/wrong information here. Just saying "It doesn't work" without details helps no one. - Yes, I still complained about GPIO, but details would be for a different issue, probably in the according git-project.

Gadgetoid commented 3 months ago

Just saying "It doesn't work" helps no one.

I appreciate the solidarity, if nothing else 😭

@kingosticks afaik --system-site-packages or BREAK_SYSTEM_PACKAGES are roughly equivalent since Python packages will "shadow" system ones when you're in a venv context and cause some of the (imagined afaict) problems that this whole debacle aims to avoid.

The benefit of --system-site-packages over BREAK_SYSTEM_PACKAGES (the latter causes pip to just disregard the warnings) is that packages will only appear broken (if they do at all) when in a venv context. It's less breaky, so if you can wrangle running Mopidy in the virtual env then it will be self-contained.

Downside of managing in a venv is that Mopidy seems to prefer apt packaging? The two mix poorly, even with --system-site-packages since a Python project cannot specify an apt dependency- if your apt/pypi stuff is out of sync then deps pulling into the venv will shadow the system packages and cause confusion.

I'm probably preaching to the choir here, though. And I'm not even really sure I'm answering the question you asked, or any question at all.

kingosticks commented 3 months ago

@kingosticks afaik --system-site-packages or BREAK_SYSTEM_PACKAGES are roughly equivalent since Python packages will "shadow" system ones when you're in a venv context and cause some of the (imagined afaict) problems that this whole debacle aims to avoid.

I'm not sure that's an entirely fair distinction. The --system-site-packages venv option can never break your system. The --break-system-packages pip option (or the equivalent env variable also mentioned) technically could. I have never experienced it either but I appreciate how it could happen.

To distribute a system service on Debian, apt is the most sensible way. You can't really distribute a Debian system service via PyPi. We also need a way to distribute our non-Python dependency for Mopidy-Spotify, apt is how we do that. And once you've got something on apt, there's a low barrier to putting everything you can on there and enjoying the benefits of system package management. Before the changes, this all used to work pretty well (in my biased opinion). Despite the current Python-world mess, I still think apt is the best way to install as much of Mopidy as possible.

since a Python project cannot specify an apt dependency

Technically it can specify and enforce the dependency. It just can't resolve it properly and yes, you get the shadowing situation. We use this flow for our development environments and it mostly works OK. However, I do agree with you and I'm not sure I would burden regular users with this more fragile/confusing setup.

whimshot commented 1 month ago

The notice for this says that the issue has been solved. However when I attempt to run the install.sh script a rather long error. I'm doing this on a Raspberry Pi 400 running the latest Raspi OS 64bit (Bookworm)

$ uname -a
Linux rpi400 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux

Here is the output from install.sh

sudo ./install.sh
Updating apt and installing dependencies
Hit:1 https://apt.mopidy.com bullseye InRelease
Hit:2 http://deb.debian.org/debian bookworm InRelease
Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
Hit:4 http://deb.debian.org/debian bookworm-updates InRelease
Hit:5 http://archive.raspberrypi.com/debian bookworm InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3-rpi.gpio is already the newest version (0.7.1~a4-1+b4).
python3-spidev is already the newest version (20200602~200721-1+bookworm).
python3-pip is already the newest version (23.0.1+dfsg-1+rpt1).
python3-pil is already the newest version (9.4.0-1.1+deb12u1).
python3-numpy is already the newest version (1:1.24.2-1).
libopenjp2-7 is already the newest version (2.5.0-2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Verifying python 3.x version
Found Python 3.11

Backing up mopidy config to: /etc/mopidy/mopidy.conf.backup-2024-06-12-09-43-47

Installing mopidy packages
mopidy was already not on hold.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
mopidy is already the newest version (3.4.2-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Installing Iris web UI for Mopidy
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 514, in get_distribution
    dist = get_provider(dist)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 386, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 956, in require
    needed = self.resolve(parse_requirements(requirements))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 815, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 856, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'mopidy_iris' distribution was not found and is required by the application
Could not find system.sh path for mopidy_iris using python3
Refusing to edit /etc/sudoers.d/010_mopidy-nopasswd with empty system.sh path!
Installing Pirate Audio plugins...
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Resetting /etc/mopidy/mopidy.conf to package defaults.
Any custom settings have been backed up to /etc/mopidy/mopidy.conf.backup-2024-06-12-09-43-47

Configuring Mopidy

Enabling and starting Mopidy
Synchronizing state of mopidy.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable mopidy

All done!
Mopidy configuration has changed, see summary below and make sure to update /etc/mopidy/mopidy.conf!
Your previous configuration was backed up to /etc/mopidy/mopidy.conf.backup-2024-06-12-09-43-47
103,150d102
<
< [raspberry-gpio]
< enabled = true
< bcm5 = play_pause,active_low,250
< bcm6 = volume_down,active_low,250
< bcm16 = next,active_low,250
< bcm20 = volume_up,active_low,250
< bcm24 = volume_up,active_low,250
<
< [file]
< enabled = true
< media_dirs = /home/pi/Music
< show_dotfiles = false
< excluded_file_extensions =
<   .directory
<   .html
<   .jpeg
<   .jpg
<   .log
<   .nfo
<   .pdf
<   .png
<   .txt
<   .zip
< follow_symlinks = false
< metadata_timeout = 1000
<
< [pidi]
< enabled = true
< display = st7789
< rotation = 90
<
< [mpd]
< hostname = 0.0.0.0
<
< [http]
< hostname = 0.0.0.0
<
< [audio]
< mixer_volume = 40
< output = alsasink device=hw:sndrpihifiberry
<
< [spotify]
< enabled = false
< username =
< password =
< client_id =
< client_secret =

kingosticks commented 1 month ago

The notice for this says that the issue has been solved. However when I attempt to run the install.sh script a rather long error. I'm doing this on a Raspberry Pi 400 running the latest Raspi OS 64bit (Bookworm)

$ uname -a
Linux rpi400 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux

Here is the output from install.sh

sudo ./install.sh
Updating apt and installing dependencies
Hit:1 https://apt.mopidy.com bullseye InRelease
Hit:2 http://deb.debian.org/debian bookworm InRelease
Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
Hit:4 http://deb.debian.org/debian bookworm-updates InRelease
Hit:5 http://archive.raspberrypi.com/debian bookworm InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3-rpi.gpio is already the newest version (0.7.1~a4-1+b4).
python3-spidev is already the newest version (20200602~200721-1+bookworm).
python3-pip is already the newest version (23.0.1+dfsg-1+rpt1).
python3-pil is already the newest version (9.4.0-1.1+deb12u1).
python3-numpy is already the newest version (1:1.24.2-1).
libopenjp2-7 is already the newest version (2.5.0-2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Verifying python 3.x version
Found Python 3.11

Backing up mopidy config to: /etc/mopidy/mopidy.conf.backup-2024-06-12-09-43-47

Installing mopidy packages
mopidy was already not on hold.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
mopidy is already the newest version (3.4.2-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Installing Iris web UI for Mopidy
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 514, in get_distribution
    dist = get_provider(dist)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 386, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 956, in require
    needed = self.resolve(parse_requirements(requirements))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 815, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 856, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'mopidy_iris' distribution was not found and is required by the application
Could not find system.sh path for mopidy_iris using python3
Refusing to edit /etc/sudoers.d/010_mopidy-nopasswd with empty system.sh path!
Installing Pirate Audio plugins...
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Resetting /etc/mopidy/mopidy.conf to package defaults.
Any custom settings have been backed up to /etc/mopidy/mopidy.conf.backup-2024-06-12-09-43-47

Configuring Mopidy

Enabling and starting Mopidy
Synchronizing state of mopidy.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable mopidy

All done!
Mopidy configuration has changed, see summary below and make sure to update /etc/mopidy/mopidy.conf!
Your previous configuration was backed up to /etc/mopidy/mopidy.conf.backup-2024-06-12-09-43-47
103,150d102
<
< [raspberry-gpio]
< enabled = true
< bcm5 = play_pause,active_low,250
< bcm6 = volume_down,active_low,250
< bcm16 = next,active_low,250
< bcm20 = volume_up,active_low,250
< bcm24 = volume_up,active_low,250
<
< [file]
< enabled = true
< media_dirs = /home/pi/Music
< show_dotfiles = false
< excluded_file_extensions =
<   .directory
<   .html
<   .jpeg
<   .jpg
<   .log
<   .nfo
<   .pdf
<   .png
<   .txt
<   .zip
< follow_symlinks = false
< metadata_timeout = 1000
<
< [pidi]
< enabled = true
< display = st7789
< rotation = 90
<
< [mpd]
< hostname = 0.0.0.0
<
< [http]
< hostname = 0.0.0.0
<
< [audio]
< mixer_volume = 40
< output = alsasink device=hw:sndrpihifiberry
<
< [spotify]
< enabled = false
< username =
< password =
< client_id =
< client_secret =

Do https://github.com/pimoroni/pirate-audio/issues/98#issuecomment-1925717271

pimoroni / pirate-audio

mopidy/install.sh: broken on Bookworm #98