search

nRF24 / pyRF24

A python package that wraps the RF24, RF24Network, and RF24Mesh C++ libraries.
http://pyrf24.rtfd.io
GNU General Public License v2.0
24 stars 3 forks source link

uploading binary wheels to pypi #6

Closed 2bndy5 closed 2 years ago

2bndy5 commented 2 years ago

I have been researching how to upload pre-built binary wheels to pypi... We cannot upload a source distributionto pypi because this pkg is built using submodules, and pip has no support (& never will) for installing from a source that uses submodules. For example, using pip install git+https://github.com/nRF24/pyRF24.git will fail because the submodules are not initialized when pip clones the repo into a temp env/workspace.

I'm using submodules to isolate the built pyrf24 pkg from the built/installed C++ equivalent libs. This improves portability and allows us to pin the pkg to a specific commit/tag of the RF24* libs.

Currently, there are 2 problems that hinder the process of uploading wheels that will work on various Linux SoC devices:

  1. The C standard lib version (older has better compatibility) and python3 version (specifically the python3-dev package available from the OS's PPA via apt).
  2. The hardware's CPU architecture (including bitness). RPi2 through RPi4 all use armv7l while RPi0 and RPi1 use armv6.

I have been looking heavily at using docker containers to cross-build the pkg for distribution, but it looks like we may need a VM (like QEMU) to emulate some of the hardware's bitness.

I think that I'll have to setup an RPi0 and RPi3 to build wheels targeting the RPi OS (32 bit), then manually publish the pyrf24 package with the resulting built wheels. It may be easier to cross-build 64 bit binary wheels, but I'm more concerned with 32 bit since RPi OS is still very focused on using 32 bit OS until the market share of RPi1 goes extinct.

In the meantime, people will have to build this pkg from source like so

git clone --recurse-submodules https://github.com/nRF24/pyRF24.git
cd pyRF24
python -m pip install .

Although, using python setup.py install will have more verbose output than the using pip install .. Either way will take about 5 minutes to build/install depending on the device's resources.

2bndy5 commented 2 years ago

Well, I tried out the pyps's cibuildwheel action on the update-submodules branch, and it worked... I'm not sure if any of the built binary dists are compatible with the armhf arch yet (unlikely). The workflow with the cibuildwheel action using its defaults (+ aarch64 builds) took over 2 hours to complete. So, I'm thinking we can use it to build only aarch64 and manylinux wheels for python3.7-3.10 (& maybe pypy37-39 if applicable) to reduce the build time. I see this as a huge step toward uploading binary dists to pypi.org.

For any platform not directly targeted, I'm looking into uploading a source dist to pypi (prompting pip to build the pkg from source).

2bndy5 commented 2 years ago

So, none of the wheels built by cibuildwheels are compatible with armhf (aka armv6l or armv7l). However, there is a piwheels project that aims to create their own index of binary wheels targeting the Raspberry Pi OS (but not 64 bit which we can build with cibuildwheels).

For now, I'm moving forward in hopes that piwheels project will see our new releases on pypi and build a wheel that users can install from the piwheels index (which is already added to the list of indexes that pip looks for in RPi OS images).

2bndy5 commented 2 years ago

I think I can skip building musllinux bdist wheels to reduce the workflow run time (which should cut it down to ~1 hour).

I just tried installing from pypi on Ubuntu 64 bit (on my RPI 4), and pip defaulted to the aarch64 bdist wheel using the manylinux tag.

TMRh20 commented 2 years ago

Working fine on RPi model B, RPi3 and RPi4 here w/RPI OS. FYI, on one of the RPi model b if gave an error: FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/pip3.9' So i just created a link to /usr/bin/pip3 and it worked...

2bndy5 commented 2 years ago

I had to do a lot of work on Ubuntu:

I also tested building the pkg using RPi driver which required installing

Using sudo for the RPi driver was fine, but I use a python venv which required explicitly specifying the venv's python executable:

(env) brendan@rpi4-ubuntu-64bit:~/github/pyRF24$ sudo ~/env/bin/python examples/scanner.py

I keep getting an error when building with the MRAA driver

undefined symbol: mraa_gpio_use_mmaped_internal

This seems consistent with C++ installs. Not yet sure what is going wrong there - maybe they removed a deprecated function.

The MRAA definition in src: https://github.com/eclipse/mraa/blob/046bdd0adbd0c7259ee0bd19e01091c25f043cde/api/mraa/gpio.h#L280-L292