How best to install rawpy on (32-bit) Raspberry Pi Zero / Raspbian?

[greiginsydney]

Hi @letmaik , apologies for what's probably a dumb/simple question, but I'm struggling with the 'right' way to install rawpy on my 32-bit Raspberry Pi Zero.

If I sudo pip3 install rawpy I end up with:

Successfully installed rawpy-0.13.1

... which is way old. Is that just going with the latest wheel it can find for the Pi, or am I trying the wrong way? (I'm using pip 21.1.3).

(I note your comment on Issue #145 about building from source - which has been my practice thus far - but also a reference to seemingly fixing(?) this in Issue #34 ).

Ta for rawpy, BTW.

[letmaik]

PyPI does not have Raspberry PI compatible wheels at all. What you're getting is the latest version of rawpy that had an sdist (.tar.gz source code) as part of its release. It then used that and compiles from source. The reason why newer releases don't have sdists anymore is that building from source typically won't work and leads to weird errors (esp. on Windows) and more bug reports. Ideally, I'd like to remove all sdists from older versions on PyPI, but I don't think messing with old releases is a good idea.

To be more concrete why sdists are a pain, the default policy of pip install is something like:

• if the latest rawpy version has a compatible wheel, use it
• otherwise, if the latest version has an sdist, build from source
• otherwise, try to find an older version which either has a compatible wheel or an sdist and either use the wheel or build from source, whatever comes first

This means, any time I want to remove support for a certain platform (like Windows 32 bit), I simply don't produce wheels anymore and users who for some reason still use Python 32 bit will happily install the last supported version via the wheel. If an sdist was published as well, then pip would try to build it from source instead, and fail. This can be prevented with --prefer-binary when doing pip install but realistically no one would think of that because the error message from a failed source build is way too cryptic.

There is https://www.piwheels.org which builds non-standard Pi wheels that are not on PyPI, but they require an sdist for their automation.

Given that you just want an easy way to install without manual cloning etc. I think the following is the closest:

pip install cython numpy
pip install git+https://github.com/letmaik/rawpy.git@v0.16.0

[letmaik]

Side note: I decided to remove all old sdists of rawpy as they keep causing trouble. I will try to improve the build script so that a separate install of cython and numpy won't be necessary, and then it should be useable enough with the install-from-git method if someone needs it.

[greiginsydney]

Thanks!

[Bra1nsen]

pi@skycamuni:~/RAW $ sudo pip install cython numpy Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Collecting cython Using cached https://www.piwheels.org/simple/cython/Cython-0.29.32-cp39-cp39-linux_armv7l.whl (5.0 MB) Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (1.19.5) Installing collected packages: cython Successfully installed cython-0.29.32 pi@skycamuni:~/RAW $ pip install git+https://github.com/letmaik/rawpy.git@v0.16.0 Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Collecting git+https://github.com/letmaik/rawpy.git@v0.16.0 Cloning https://github.com/letmaik/rawpy.git (to revision v0.16.0) to /tmp/pip-req-build-bjxf9hob Running command git clone -q https://github.com/letmaik/rawpy.git /tmp/pip-req-build-bjxf9hob Running command git checkout -q 5e510fbc5f7b75f99380bd71d83b3d3a7db06888 Running command git submodule update --init --recursive -q fatal: unable to connect to github.com: github.com[0: 140.82.121.3]: errno=Connection timed out

fatal: clone of 'git://github.com/LibRaw/LibRaw.git' into submodule path '/tmp/pip-req-build-bjxf9hob/external/LibRaw' failed Failed to clone 'external/LibRaw'. Retry scheduled fatal: unable to connect to github.com: github.com[0: 140.82.121.3]: errno=Connection timed out

fatal: clone of 'git://github.com/LibRaw/LibRaw-cmake' into submodule path '/tmp/pip-req-build-bjxf9hob/external/LibRaw-cmake' failed Failed to clone 'external/LibRaw-cmake'. Retry scheduled fatal: unable to connect to github.com: github.com[0: 140.82.121.4]: errno=Connection timed out

fatal: clone of 'git://github.com/LibRaw/LibRaw-demosaic-pack-GPL2.git' into submodule path '/tmp/pip-req-build-bjxf9hob/external/LibRaw-demosaic-pack-GPL2' failed Failed to clone 'external/LibRaw-demosaic-pack-GPL2'. Retry scheduled

[svep-design-center]

For anyone stumbling across thiss in the future, I managed to get it working by doing the following:

sudo apt install libraw-dev
python3 -m pip install cython
python3 -m pip install git+https://github.com/letmaik/rawpy.git@v0.17.3

[greiginsydney]

Hello again @letmaik .

I've tried all of the suggested ways of installing rawpy on Raspberry Pi OS Bookworm / Debian 12.4 without success.

• installation-from-source-on-linuxmacos
• Your suggestion above

A few variables have changed, including this is python 3.11.2, and I'm building into a virtual environment, but regardless of the process I try I still hit the same error:

Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Processing /home/pi/libraw/rawpy
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "/home/pi/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/pi/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/pi/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qowkmkdr/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qowkmkdr/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-qowkmkdr/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 480, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-qowkmkdr/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 11, in <module>
      ModuleNotFoundError: No module named 'numpy'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

I'm struggling to understand if the ModuleNotFoundError: No module named 'numpy' is the root cause or a red herring, given the earlier messages - and of course numpy is already installed:

pip3 show numpy
Name: numpy
Version: 1.26.3
<edit>
Location: /home/pi/venv/lib/python3.11/site-packages
Requires:
Required-by: imageio

Does anything in the above hint at the issue, or is there some other change perhaps related to 3.11 and/or 32-bit architectures? My hardware platform is the Pi Zero W, which is a 32-bit ARMv6 CPU.

- G.