Add required runtime package dependancies.

[WayneKeenan]

I thought this felt a bit deja-vu... https://github.com/ukBaz/python-bluezero/issues/111

I installed into a venv (without --system-site-packages) so these packages were missing.
Which reminds me of another ticket...

I guess this can be dropped (?)

[WayneKeenan]

Actually, perhaps it could just be a requirements.txt file with the 3 reps and a documentation note as why it's there.

[ukBaz]

The problem that I grapple with on this one is that the following three packages are not fully install-able by pip from PyPI

required_packages = ['dbus-python', 'gi', 'PyGObject']

Or at least they require system packages to be installed for these to work. As mentioned elsewhere these system packages come as default on most(?) Linux systems. If you are running without a venv then these PyPI packages are not required as the packages are available from the system packages.

There is then the specific requirement of getting Bluezero to work in a venv.

For the last two I thought this was covered by:

https://pypi.org/project/vext.gi/

While this package is not required for a non-venv install.

For dbus, I thought it just didn't work inside of a venv at all. However, reading the following it would suggest there is a fix for this to work in a venv:

https://bugs.freedesktop.org/show_bug.cgi?id=55439

This has left me all a little uncertain as to what is the best thing to document.

My thought is the default case the tool should support is what is required for installation on a Raspberry Pi without a venv.

There should then be documentation of how to install Bluezero into a venv.

Given my knowledge seems to be out, do we know what steps are required to install on a RPi with and without a venv?

If we don't, then I'll do a fresh install on a RPi and research this.

Thanks.

[WayneKeenan]

This venv setup without site packages and installation with pip got bluezero installed and working for me:

python3 -m pip venv venv
. ./venv/bin/activate
pip install -r requirements.txt

requirements.txt:

dbus-python
gi
PyGObject
bluezero

Maybe I should back and do it on a fresh Raspian install.
I'm using stretch and buster as other non-BLE things changed too much with bullseye.

How is BlueZ with bullseye these days? Are you targeting it?

[WayneKeenan]

I've been installing from scratch (fresh OS) on Buster...

The gi is not needed in requirements.txt.

I've asked (tweeted) to Ben to add PyGObject to PiWheels as this is the only reason the 3 deb dev packages (libglib2.0-dev libgirepository1.0-dev libcairo2-dev) need to be installed when using a venv with no site packages.

[WayneKeenan]

For Buster:

Steps in a venv (with site packages)

python3 -m venv venv_site --system-site-packages
. ./venv_site/bin/activate
pip install bluezero
git clone https://github.com/ukBaz/python-bluezero.git
python python-bluezero/examples/heartrate_monitor_peripheral.py
Advertisement registered

Steps in a venv (no site packages)

python3 -m venv venv_nosite
. ./venv_nosite/bin/activate
sudo apt install -y libglib2.0-dev libgirepository1.0-dev libcairo2-dev
pip install dbus-python PyGObject bluezero
git clone https://github.com/ukBaz/python-bluezero.git
python python-bluezero/examples/heartrate_monitor_peripheral.py
Advertisement registered

[WayneKeenan]

Request to add PyGObject to PiWheels: https://github.com/piwheels/packages/issues/336

[WayneKeenan]

The steps above for site and no-site work for Bullseye too.

[WayneKeenan]

On Bullseye 2022-09-22 (no venv)

Version info: pip 20.3.4 from /usr/lib/python3/dist-packages/pip (python 3.9)

Running:

pip install bluezero
git clone https://github.com/ukBaz/python-bluezero.git
python python-bluezero/examples/heartrate_monitor_peripheral.py
Advertisement registered

Installs into : ~/.local/lib/python3.9/site-packages

Because pip has some (configurable) smarts so in this case it is defaulting to a --user install and won't go into the write protected system package folder.

Running:

sudo pip install bluezero

Installs into /usr/local/lib/python3.9/dist-packages

Which would be more suitable if your writing a bluezero based script intended to run under a dedicated user as a system service (and not using a dedicated venv for the service)

[WayneKeenan]

I overlooked your vext.gi comment, I think because of some painful memory that was actually todo with gi... so...

It would have saved me a lot of time if I hadn't, as that works fine:

python -m venv venv
. ./venv/bin/activate
pip install dbus-python
pip install vext.gi
pip install bluezero
git clone https://github.com/ukBaz/python-bluezero.git
python python-bluezero/examples/heartrate_monitor_peripheral.py
Advertisement registered

[ukBaz]

I have been thinking about this and my concern with the current PR is that it has dbus-python and vext.gi as required packages.

e.g.

required_packages = ['dbus-python', 'vext.gi']

I don't think they are required packages unless you are installing in to a venv.

Would it make more sense to define a new install group (e.g. venv) such as:

required_packages = []
extras_venv = ['dbus-python', 'vext.gi']
extras_rel = ['bumpversion', 'twine']
extras_doc = ['sphinx', 'sphinx_rtd_theme']
extras_test = ['coverage', 'pycodestyle', 'python-dbusmock']
extras_dev = extras_rel + extras_doc + extras_test
...
    extras_require={
        'rel': extras_rel,
        'venv': extras_venv,
        'docs': extras_doc,
        'test': extras_test,
        'dev': extras_dev,
    },

Then to install in a venv it would be

pip install bluezero[venv]

It would still need to be documented, but I don't know of any other way to get a different install for a venv versus a "normal" install.

Thoughts?

[WayneKeenan]

But they are dependancies of bluezero, inside or outside of a venv. On a "normal" install those dependancies would be detected and so nothing would be done? (Untested)

[ukBaz]

Sorry to keep going round on this but I'm concerned we don't quite have the details correct on this for all the different scenarios.

The only(?) requirement for Bluezero is that the system has Python bindings for D-Bus and GLib. These are typically installed on a machine (especially RPi) already. If those packages are not, then the normal flow would be to install them as apt packages which would get all the other dependencies to.

Confirming these are already installed on my RPi:

$ apt list --installed | grep dbus | grep python
python3-dbus/stable,now 1.2.16-5 armhf [installed,automatic]

$ apt list --installed | grep gi | grep python
python3-gi/stable,now 3.38.0-2 armhf [installed]

Looking at how the system would react if the apt packages done by the user if already installed:

$ sudo apt install python3-dbus
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3-dbus is already the newest version (1.2.16-5).
python3-dbus set to manually installed.
The following package was automatically installed and is no longer required:
  libfuse2
Use 'sudo apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 234 not upgraded.

$ sudo apt install python3-gi
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3-gi is already the newest version (3.38.0-2).
The following packages were automatically installed and are no longer required:
  cups-pk-helper gir1.2-notify-0.7 gir1.2-packagekitglib-1.0
  gir1.2-polkit-1.0 gir1.2-secret-1 hplip-data libfuse2 libhpmud0
  libsane-hpaio printer-driver-hpcups printer-driver-postscript-hp
  python3-cups python3-pexpect python3-ptyprocess python3-renderpm
  python3-reportlab python3-reportlab-accel python3-smbc
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 234 not upgraded.

Looking to see what would happen if the two dependencies were specified in the setup.py:

$ sudo pip3 install dbus-python
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: dbus-python in /usr/lib/python3/dist-packages (1.2.16)
$ python3 -c "import dbus; print(dbus.__file__)"
/usr/lib/python3/dist-packages/dbus/__init__.py
$ pip list | grep dbus
dbus-python                   1.2.16
$ sudo pip3 uninstall dbus-pyhton
WARNING: Skipping dbus-pyhton as it is not installed.

Specifying PyOBject as a pip install doesn't work:

$ sudo pip3 install PyOBject
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting PyOBject
  Downloading pyobject-1.1.3.tar.gz (18 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ah5j6icl/pyobject_16df1f9856924b13b0cab496f370eff1/setup.py", line 8, in <module>
          long_desc=open("README.rst").read()
        File "/usr/lib/python3.9/codecs.py", line 322, in decode
          (result, consumed) = self._buffer_decode(data, self.errors, final)
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb8 in position 13: invalid start byte
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Although it is there:

$ python3 -c "from gi.repository import GLib;print('success')"
success

To paraphrase the install for PyGObject from the documentation as I think it relates to Bluezero:

Installing the system provided PyGObject: 1) Open a terminal 2) Execute sudo apt install python3-gi

Installing from PyPI with pip: 1) Open a terminal and enter your virtual environment 2) Execute sudo apt install libgirepository1.0-dev gcc libcairo2-dev pkg-config python3-dev gir1.2-gtk-3.0 to install the build dependencies and GTK 3) Execute pip3 install PyGObject to build and install PyGObject

I think to specify this in a similar way for D-Bus it would be:

Installing the system provided D-Bus: 1) Open a terminal 2) Execute sudo apt install python3-dbus

Installing from PyPI with pip: 1) Open a terminal and enter your virtual environment 2) Execute sudo apt install libdbus-glib-1-dev libdbus-1-dev to install the build dependencies and D-Bus 3) Execute pip3 install dbus-python to build and install D-Bus for Python3

To run the experiment of adding these packages to a venv:

$ python3 -m venv venv
$ . venv/bin/activate
(venv) $ pip install pip --upgrade
[...]
(venv) $ python -c "import dbus"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'dbus'
(venv) $ pip install dbus-python
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting dbus-python
  Downloading https://www.piwheels.org/simple/dbus-python/dbus_python-1.3.2-cp39-cp39-linux_armv7l.whl (114 kB)
     |████████████████████████████████| 114 kB 4.5 MB/s 
Installing collected packages: dbus-python
Successfully installed dbus-python-1.3.2
(venv) $ python -c "import dbus"
(venv) $

(venv) $ python -c "from gi.repository import GLib"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'gi'
(venv) $ sudo apt install libgirepository1.0-dev gcc libcairo2-dev pkg-config python3-dev gir1.2-gtk-3.0
(venv) $ pip install PyGObject
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting PyGObject
  Using cached PyGObject-3.42.2.tar.gz (719 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pycairo>=1.16.0
  Using cached https://www.piwheels.org/simple/pycairo/pycairo-1.23.0-cp39-cp39-linux_armv7l.whl (296 kB)
Building wheels for collected packages: PyGObject
  Building wheel for PyGObject (pyproject.toml) ... done
  Created wheel for PyGObject: filename=PyGObject-3.42.2-cp39-cp39-linux_armv7l.whl size=728411 sha256=e7799aa695a6d7d61251a467b4923fb60d4a991045a1d5032100b2c813284aa9
  Stored in directory: /home/pi/.cache/pip/wheels/c4/2e/6f/761793b8a4705cc4495bf6db1e1ac174526a5e30f272633855
Successfully built PyGObject
Installing collected packages: pycairo, PyGObject
Successfully installed PyGObject-3.42.2 pycairo-1.23.0
(venv) $ python -c "from gi.repository import GLib"
(venv) $ 

Removing the python3-dbus package and try to install it with pip:

$ sudo apt remove python3-dbus
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  cups-pk-helper gir1.2-notify-0.7 gir1.2-packagekitglib-1.0
  gir1.2-polkit-1.0 gir1.2-secret-1 hplip-data libfuse2 libhpmud0
  libsane-hpaio printer-driver-hpcups printer-driver-postscript-hp
  python3-cups python3-pexpect python3-ptyprocess python3-renderpm
  python3-reportlab python3-reportlab-accel python3-smbc
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  hplip python3-cupshelpers python3-dbus system-config-printer
  system-config-printer-common system-config-printer-udev
0 upgraded, 0 newly installed, 6 to remove and 234 not upgraded.
After this operation, 7,193 kB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 105410 files and directories currently installed.)
Removing hplip (3.21.2+dfsg1-2+b1) ...
Removing system-config-printer (1.5.14-1) ...
Removing system-config-printer-udev (1.5.14-1) ...
Removing system-config-printer-common (1.5.14-1) ...
Removing python3-cupshelpers (1.5.14-1) ...
Removing python3-dbus (1.2.16-5) ...
Processing triggers for gnome-menus (3.36.0-1) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for dbus (1.12.20-2) ...
Processing triggers for mailcap (3.69) ...
Processing triggers for desktop-file-utils (0.26-1) ...
pi@SensePi:~/venv_tests $ python3 -c "import dbus; print(dbus.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'dbus'
$ sudo pip3 install dbus-python
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting dbus-python
  Downloading https://www.piwheels.org/simple/dbus-python/dbus_python-1.3.2-cp39-cp39-linux_armv7l.whl (114 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.3/114.3 kB 164.4 kB/s eta 0:00:00
Installing collected packages: dbus-python
Successfully installed dbus-python-1.3.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
$ python3 -c "import dbus; print(dbus.__file__)"
/usr/local/lib/python3.9/dist-packages/dbus/__init__.py
$ sudo pip3 uninstall dbus-python
Found existing installation: dbus-python 1.3.2
Uninstalling dbus-python-1.3.2:
  Would remove:
    /usr/local/include/python3.9/dbus-python/dbus-1.0/dbus/dbus-python.h
    /usr/local/lib/python3.9/dist-packages/.dbus_python.mesonpy.libs/pkgconfig/dbus-python.pc
    /usr/local/lib/python3.9/dist-packages/_dbus_bindings.cpython-39-arm-linux-gnueabihf.so
    /usr/local/lib/python3.9/dist-packages/_dbus_glib_bindings.cpython-39-arm-linux-gnueabihf.so
    /usr/local/lib/python3.9/dist-packages/dbus/*
    /usr/local/lib/python3.9/dist-packages/dbus_python-1.3.2.dist-info/*
Proceed (Y/n)? 
  Successfully uninstalled dbus-python-1.3.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
$ python3 -c "import dbus; print(dbus.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'dbus'
$ sudo apt install python3-dbus
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  cups-pk-helper gir1.2-notify-0.7 gir1.2-packagekitglib-1.0
  gir1.2-polkit-1.0 gir1.2-secret-1 hplip-data libfuse2 libhpmud0
  libsane-hpaio printer-driver-hpcups printer-driver-postscript-hp
  python3-cups python3-pexpect python3-ptyprocess python3-renderpm
  python3-reportlab python3-reportlab-accel python3-smbc
Use 'sudo apt autoremove' to remove them.
Suggested packages:
  python-dbus-doc python3-dbus-dbg
The following NEW packages will be installed:
  python3-dbus
0 upgraded, 1 newly installed, 0 to remove and 234 not upgraded.
Need to get 100 kB of archives.
After this operation, 386 kB of additional disk space will be used.
Get:1 http://raspbian.mirror.uk.sargasso.net/raspbian bullseye/main armhf python3-dbus armhf 1.2.16-5 [100 kB]
Fetched 100 kB in 5s (18.5 kB/s) 
Selecting previously unselected package python3-dbus.
(Reading database ... 105124 files and directories currently installed.)
Preparing to unpack .../python3-dbus_1.2.16-5_armhf.deb ...
Unpacking python3-dbus (1.2.16-5) ...
Setting up python3-dbus (1.2.16-5) ...
$ python3 -c "import dbus; print(dbus.__file__)"
/usr/lib/python3/dist-packages/dbus/__init__.py

In summary:

I think it comes down to this table depending if you are using a venv or not:

	No apt pkg	no py pkg
no venv	$ sudo apt install python3-dbus $ sudo apt install python3-gi
with venv	$ sudo apt install libglib2.0-dev libgirepository1.0-dev gcc libcairo2-dev \ pkg-config python3-dev gir1.2-gtk-3.0	$ pip install dbus-python $ pip install PyOBject

I've opened the following ticket to help with the install process. https://github.com/piwheels/packages/issues/339

My belief (as I have no way of proving this) is that 1) most people are working without a venv. 2) most systems have the required apt modules installed 3) There is no PyPI package that can be put in the setup.py file that will work with (and without) a venv.

As a result I don't want to do anything that will break the non-venv case. In the short term the solution may be to document the additional steps to setup a venv correctly.

Any such documentation will should also note that using --system-site-packages is an option.

In my opinion this is not a great situation to be in. It is caused by the fact that GObject and D-Bus python libs are wrappers around low-level (non-python) packages which makes this not tidy to work with in Python.

[ukBaz]

I think this PR has been useful because of the information captured in it. However, I don't think it is ever going to get merged based on my last note so I'm going to close this. Thanks Wayne.