[BUG] Cannot be used with non-system python

[Kasperror]

Describe the bug Running import picamera2 under python that is non-system python rasises ModuleNotFoundError: No module named 'libcamera'. This is because dependency installation python3-libcamera creates python binding only for system python, rendering development of applications using picamera2 limited to current Pi OS base python version.

To Reproduce first install all dependencies according to README.md then:

virtualenv venv --python /usr/local/bin/python3.11
source venv/bin/activate
pip install picamera2
python

from within python repl

import picamera2

which rasises

  File "<stdin>", line 1, in <module>
  File "/home/pi/libcamera/venv/lib/python3.11/site-packages/picamera2/__init__.py", line 1, in <module>
    from .configuration import CameraConfiguration, StreamConfiguration
  File "/home/pi/libcamera/venv/lib/python3.11/site-packages/picamera2/configuration.py", line 1, in <module>
    from .controls import Controls
  File "/home/pi/libcamera/venv/lib/python3.11/site-packages/picamera2/controls.py", line 4, in <module>
    from libcamera import ControlType, Size, Rectangle
ModuleNotFoundError: No module named 'libcamera'

Expected behaviour Default recommended installation with pip with python bindings created for the python connected with pip. This would make isolated development on any python version possible.

Console Output, Screenshots If applicable, any console output or screenshots that show the problem and associated error messages.

Hardware : Raspberry Pi 4B running Pi OS Bullseye 64-bit

[davidplowman]

Hi, thanks for the question. You probably need your virtual environment to copy the system packages because not all are available via pip, for example:

python -m venv --system-site-packages my-env

[Kasperror]

Hi, thanks for the question. You probably need your virtual environment to copy the system packages because not all are available via pip, for example:

python -m venv --system-site-packages my-env

This won't work since the --system-site-packages adds the packages from the python version that is used inside virtualenv This solved #341 since venv package creates only isolated packages and still uses system python virtualenv on the other hand creates an isolated packages and python environment.

I've confirmed that it won't work on a clean Pi OS 64-bit with source-compiled python 3.10

BUT to actually check if you can use picamera2 with python different that which the system provides I've done a simple test

1. installed clean Pi OS 64-bit
2. updated & upgraded
3. upgraded pip for system python
4. installed python 3.10.8 from source
5. upgraded pip for installed python
6. run a sanity check on system python without virtualenv

Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import picamera2
>>> picamera2.__file__
'/usr/lib/python3/dist-packages/picamera2/__init__.py'
>>> import libcamera
>>> libcamera.__file__
'/usr/lib/python3/dist-packages/libcamera/__init__.py'

7. created and virtualenv for system python with said flag

   pip install virtualenv
   virtualenv venv --system-site-packages 
   source venv/bin/activate
   python

8. checked if the proposed works for system python (it works)

   Python 3.9.2 (default, Feb 28 2021, 17:03:44)
   [GCC 10.2.1 20210110] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import picamera2
   >>> picamera2.__file__
   '/usr/lib/python3/dist-packages/picamera2/__init__.py'
   >>> import libcamera
   >>> libcamera.__file__
   '/usr/lib/python3/dist-packages/libcamera/__init__.py'

9. deactivated the virtualenv
10. created a virtualenv for python3.10

    virtualenv venv10 --system-site-packages --python /usr/local/bin/python3.10
    source venv10/bin/activate
    python

11. checked for libcamera

    Python 3.10.8 (main, Nov 26 2022, 04:52:14) [GCC 10.2.1 20210110] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import libcamera
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'libcamera'

12. checked for the same without virtualenv (same error)

SO No matter what, the libcamera would have to be prepackaged and build duing picamera2 build. Otherwise the users are forced to install libcamera on their own from source which I personally find toubling as when buying Raspberry Pi camera v2 I was hoping for a more native and easier way to get it up and running on newer python versions. Now the only hope is the python-opencv

Please consider chaning the default installation to pip as it supposed to be in the python community. Im happy to help and run some more tests if you'd like. Im desperate to get it running on picamera2 and at least python3.10

[davidplowman]

Yes, we can look into this, though I'm afraid I can't give any timescales at the moment. But thank you for the update!

[davidplowman]

Out of interest, the python libcamera bindings are normally found here: /usr/lib/python3/dist-packages/libcamera. If you rename the .so file in there to _libcamera.so, are you able to import that successfully using Python 3.10?

[Kasperror]

I'll give you an update once I run the tests on the clean Pi (OS reinstall + installing python3.10 takes a while)

[Kasperror]

Out of interest, the python libcamera bindings are normally found here: /usr/lib/python3/dist-packages/libcamera. If you rename the .so file in there to _libcamera.so, are you able to import that successfully using Python 3.10?

I installed fresh Pi OS 64-bit along with Python3.10.9 built from source with --enable-optimizations --with-lto flags, renamed the mentioned file, rebooted, and just as suspected nothing happened. Why can't you guys package the libcamera with python bindings and post it to pypi? If you're having problems with that @davisking did a great job handling dlib bindings with pybind11 maybe he can lend a hand.

[davisking]

Thanks, I think the maintainers here can handle it without me though :)

[ttftw]

Has there been any progress on this or alternate solutions? Has anyone been able to upgrade to Python 3.10+ on a Raspberry Pi and still use Picamera2/Libcamera?

--system-site-packages does not work for me, libcamera is still not showing up in the virtual environment.

[antoine-sac]

In a virtual env (I'm using poetry like #466 but it's the same for a simple venv), the sys.path contains /usr/lib/python3.9, so we can copy the libcamera module and its dependencies there and it will be available in all virtual envs using python 3.9. It won't work if the venv is using another python version.

sudo cp -r /usr/lib/python3/dist-packages/libcamera /usr/lib/python3.9
sudo cp -r /usr/lib/python3/dist-packages/pykms /usr/lib/python3.9

I got a bunch of errors such as:

libcblas.so.3: cannot open shared object file: No such file or directory
libopenjp2.so.7: cannot open shared object file: No such file or directory

which disappeared after installing the relevant libs.

sudo apt install libatlas-base-dev libopenjp2-7-dev

It's a bit insane to have to do that and I wish I understood why this package breaks virtual envs - doesn't everyone use some sort of venv nowadays for reproducibility of deployments? That's a big deal!

But it honestly goes over my head and since I can't fix it I can't complain :)

[Kasperror]

In a virtual env (I'm using poetry like #466 but it's the same for a simple venv), the sys.path contains /usr/lib/python3.9, so we can copy the libcamera module and its dependencies there and it will be available in all virtual envs.

sudo cp -r /usr/lib/python3/dist-packages/libcamera /usr/lib/python3.9
sudo cp -r /usr/lib/python3/dist-packages/pykms /usr/lib/python3.9

I got a bunch of errors such as:

libcblas.so.3: cannot open shared object file: No such file or directory
libopenjp2.so.7: cannot open shared object file: No such file or directory

which disappeared after installing the relevant libs.

sudo apt install libatlas-base-dev libopenjp2-7-dev

It's a bit insane to have to do that and I wish I understood why this package breaks virtual envs - doesn't everyone use some sort of venv nowadays for reproducibility of deployments? That's a big deal!

But it honestly goes over my head and since I can't fix it I can't complain :)

Thanks for the solution proposal. Although its very hacky I will give it a go and test it out with pi camera 3. Does you venv use newer python version like 3.11 or sticks to the system 3.9? BTW I think since Raspberry Pi Foundation sells the cameras you actually can complain when there isn't any proper solution for a very standard use case.

[BlueWolfProduction]

Thanks for the solution proposal. Although its very hacky I will give it a go and test it out with pi camera 3. Does you venv use newer python version like 3.11 or sticks to the system 3.9? BTW I think since Raspberry Pi Foundation sells the cameras you actually can complain when there isn't any proper solution for a very standard use case.

I tried it on a fresh rpi zero 2 using pyenv to run python 3.11.3, in addition to also copying picamera2 over, and I can say it does not work. In the end I just get the same error (No module named 'libcamera._libcamera').

I think this only works when your python version is the same as your system. But then you might as well just use the --system-site-packages trick anyway.

[ttftw]

The problem that I can see is libcamera is compiled for 3.9. You can copy over picamera2 and libcamera from /usr/lib/python3/dist-packages/ to your virtual env...but the problem is libcamera won't import.

There is a file in the package called _libcamera.cpython-39-aarch64-linux-gnu.so. If you rename it to _libcamera.cpython-311-aarch64-linux-gnu.so, Python 3.11 will try to import it, but throws this error:

ImportError: Python version mismatch: module was compiled for Python 3.9, but the interpreter version is incompatible: 3.11.4 (main, Jul 6 2023, 09:14:44) [GCC 10.2.1 20210110]

I don't know enough about any of that, but is anyone able to compile this for 3.11?

[Kasperror]

@davidplowman are there any updates on this matter? We've actually downgraded the hardware from camera V3 to V2 on a 100+ devices scale since this issue forces us to work with python-opencv instead on picamera2.

[davidplowman]

Hi, I'm afraid there doesn't seem to be anyone here who knows about packaging something like this. I'm assuming it's only the libcamera Python bindings that are the issue and that everything else is OK, does that sound right? I don't know if it's possible for you to build libcamera yourself, it shouldn't be difficult, and presumably if you build it with a different version of Python then that would work for you? Sorry that I don't know anything more about this, but I would happily take advice from someone who does!

[adamgogacz]

Well, in about a week from today the Picamera2 package will be 3 Python versions behind Python3.12 release schedule. These cameras are not ready for the Python community, who by and large operate in virtual environments other than 3.9. Why complain? I now own ~$250 worth of dust collection surface.

[tvoverbeek]

@adamgogacz Raspberry Pi OS follows Debian long term (currently Debian 11). Until the next version of Debian Stable is taken up by Raspberry Pi OS do not expect the picamera2 packages being based on a Python version later than 3.9. FYI Debian 12 (Bookworm) uses python 3.11.

[adamgogacz]

@tvoverbeek I think I may have found the ugliest solution. I have my_cool_fun.py in a virtual environment with 3.11 interpreter and fun_picam3_9.py using the Picamera2 and running in the (base/system) environment. That is, fun_picam3_9.py gets launched (using Python's subprocess module) in the system (base) environment and both establish a socket connection using Python's socket module. Now, the the two functions can exchange data using Python's socket module and send whatever signals are required by the app. Crazy, eh?

[Kasperror]

@adamgogacz can't really see this as a solution. Apart from handling standard picamera2 code now you'll need to manage proper socket communication handling. Why even use the python-binded version when you can use straight up C engine and open the socket there.

[adamgogacz]

@Kasperror I got it working perfectly fine, taking photos and shipping them off to AWS. Essentially you have a socket server (python3.11 in my case) and a socket connection (python 3.9 running the picamera) doing their while True thing, using pickle to serialize the data, have server issue a 'capture a still' command to the connection, connection capture a pic, send it back to the server, and eventually all sent to AWS via MQTT. It's actually not a lot of code. My picamera code is ~100 lines with error handling, and the socket server (3.11 code) is ~100 lines as well.

Why even use the python-binded version when you can use straight up C engine and open the socket there.

Why? I threw a dart at C vs Python, landed Python (I considered assembly, but that was on the other side of my dart board).

I gotta tell you, this is a way cleaner solution than downgrading hardware. Like the socket idea?

[Kasperror]

@adamgogacz Well to be honest creating a whole background process in different interpreter version and connecting in through a socket just to achieve goal of simply capturing a frame from a camera is not on my list of clean solutions. Let's just agree to disagree and hope that shortly there won't be a problem when pi foundation considers making up-to-date software for its hardware.

[DocGarbanzo]

@adamgogacz Raspberry Pi OS follows Debian long term (currently Debian 11). Until the next version of Debian Stable is taken up by Raspberry Pi OS do not expect the picamera2 packages being based on a Python version later than 3.9. FYI Debian 12 (Bookworm) uses python 3.11.

Shouldn't you be able to build your libcamera against python 3.9, 3.10, 3.11 and 3.12 and upload all of them to pypi? This is what other python package providers are doing as well, if they need a python-version specific code of the library. Otherwise every python project that uses the camera depends on the system python.

[JooJooBee666]

Yeah, I'm going to comment here as well. But this has also been an issue for me. This is what I use all my Pi's for; camera. There are so many advantages to using a newer Python interpreter, especially for the camera stuff and all the new optimizations offered that having this saddled to a compiled library that only works with a specific version of Python seems short-sighted. I tried many of the solutions here (before I found this report) all to come to the same conclusion. I wasted SOOO many hours trying. Where is the source code for the _libcamera.cpython-39-aarch64-linux-gnu.so library located? Here? If so, can we have clear instructions on how to properly compile for the Pi against Python 3.11?

[tvoverbeek]

@JooJooBee666 Wait till the end of October. When RPi 5 will be available also Raspberry Pi OS will go to bookworm and Python 3.11. I am assuming libcamera Python bindings will also be based on Python 3.11 then for aarch64. Maybe @davidplowman can confirm this.

[davidplowman]

The Bookworm release has been announced now, so this will indeed bump Python to 3.11.

We are aware of the problems installing Python bindings for libcamera (and indeed kms++). We are planning to look into this again once the Bookworm and Pi 5 rush has settled down, though note that they aren't Raspberry Pi "owned" packages, so no promises as to what we might be able to do.

[ttftw]

I haven't installed Bookworm yet, but if I do, are you saying this isn't currently working/updated in Raspbian?

[JooJooBee666]

@davidplowman This is better news. I'm going to install this on my test unit and see how much this wrecks my crazy python camera stuff. However, I hope we don't end up in the same trap where we're stuck on Python 3.11 until the next major OS update.

[davidplowman]

I haven't installed Bookworm yet, but if I do, are you saying this isn't currently working/updated in Raspbian?

Wasn't 100% sure what you meant. There are no pip packages for libcamera and kms++ currently. Camera applications and Picamera2 should work on Bookworm, though there are some fairly significant updates pending which will add/improve functionality.

[JooJooBee666]

Unfortunately, Bookworm is a complete mess. The use of wayland was just a bad idea; it's STILL not ready. That fact that there is only one VNC server that only half-a** works should be the flashing warning light. I can't even use it, so much stuff is broken due to this that I need that I haven't even gotten to the point of trying the camera libs(which it sounds like arent working yet either), not worth my time. Folks, why are we forcing Wayland on the unsuspecting public like this? Why are we releasing this in such horrible shape? Why rush this half-baked OS release out to the public? The issues with it are still too large to brush under the rug. :cry:

[linony0]

ModuleNotFoundError: No module named 'libcamera' Has this problem been solved?

[kneave]

ModuleNotFoundError: No module named 'libcamera' Has this problem been solved?

It seems not, I'm trying to use picamera2 from a conda environment and seeing the same, currently trying to understand how to [expletive deleted] build libcamera at the minute.

[lurch]

See https://github.com/raspberrypi/picamera2/issues/503#issuecomment-1861034147