urllib camera numpy issue

[iqgmoritz]

Describe the problem/bug

The urllib camera always fails when trying to capture a still image and the web-UI shows the error: "Could not acquire image"

Versions:

• Mycodo Version: 8.15.12
• Raspberry Pi Version: 4B
• Raspbian OS Version: RaspberryPi OS (Legacy) Lite

Reproducibility

Please list specific setup details that are involved and the steps to reproduce the behavior:

1. Go to the Camera Menu
2. Add a Camera with the urllib library
3. Install dependencies
4. Try to capture a still image (standard url or any other)
5. Error shows up

The log shows this:

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 130, in from numpy.config import show as show_config File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/config.py", line 4, in from numpy.core._multiarray_umath import ( File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/init.py", line 50, in raise ImportError(msg) ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

• The Python version is: Python3.9 from "/var/mycodo-root/env/bin/python"
• The NumPy version is: "1.26.1"

and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help.

Original error was: libopenblas.so.0: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/var/mycodo-root/mycodo/controllers/controller_function.py", line 79, in loop self.run_function.loop() File "/home/moritz/Mycodo/mycodo/functions/equation_multi.py", line 167, in loop last_measurement_a = self.get_last_measurement( File "/var/mycodo-root/mycodo/abstract_base_controller.py", line 533, in get_last_measurement return get_last_measurement(device_id, measurement_id, max_age=max_age) File "/var/mycodo-root/mycodo/utils/influx.py", line 322, in get_last_measurement last_measurement = read_influxdb_single( File "/var/mycodo-root/mycodo/utils/influx.py", line 372, in read_influxdb_single data = query_string( File "/var/mycodo-root/mycodo/utils/influx.py", line 301, in query_string ret_value = query_flux( File "/var/mycodo-root/mycodo/utils/influx.py", line 185, in query_flux from influxdb_client import InfluxDBClient File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/init.py", line 382, in from influxdb_client.client.delete_api import DeleteApi File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/delete_api.py", line 7, in from influxdb_client.client._base import _BaseDeleteApi File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/_base.py", line 22, in from influxdb_client.client.write.dataframe_serializer import DataframeSerializer File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/write/dataframe_serializer.py", line 12, in from influxdb_client.client.write.point import _ESCAPE_KEY, _ESCAPE_STRING, _ESCAPE_MEASUREMENT, DEFAULT_WRITE_PRECISION File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/write/point.py", line 40, in import numpy as np File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 135, in raise ImportError(msg) from e ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python interpreter from there. 2023-11-08 19:06:41,678 - ERROR - mycodo.controllers.controller_function_2ce6801e - Exception while running loop() Traceback (most recent call last): File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/init.py", line 24, in from . import multiarray File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/multiarray.py", line 10, in from . import overrides File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/overrides.py", line 8, in from numpy.core._multiarray_umath import ( ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 130, in from numpy.config import show as show_config File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/config.py", line 4, in from numpy.core._multiarray_umath import ( File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/core/init.py", line 50, in raise ImportError(msg) ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

• The Python version is: Python3.9 from "/var/mycodo-root/env/bin/python"
• The NumPy version is: "1.26.1"

and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help.

Original error was: libopenblas.so.0: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/var/mycodo-root/mycodo/controllers/controller_function.py", line 79, in loop self.run_function.loop() File "/home/moritz/Mycodo/mycodo/functions/equation_multi.py", line 167, in loop last_measurement_a = self.get_last_measurement( File "/var/mycodo-root/mycodo/abstract_base_controller.py", line 533, in get_last_measurement return get_last_measurement(device_id, measurement_id, max_age=max_age) File "/var/mycodo-root/mycodo/utils/influx.py", line 322, in get_last_measurement last_measurement = read_influxdb_single( File "/var/mycodo-root/mycodo/utils/influx.py", line 372, in read_influxdb_single data = query_string( File "/var/mycodo-root/mycodo/utils/influx.py", line 301, in query_string ret_value = query_flux( File "/var/mycodo-root/mycodo/utils/influx.py", line 185, in query_flux from influxdb_client import InfluxDBClient File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/init.py", line 382, in from influxdb_client.client.delete_api import DeleteApi File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/delete_api.py", line 7, in from influxdb_client.client._base import _BaseDeleteApi File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/_base.py", line 22, in from influxdb_client.client.write.dataframe_serializer import DataframeSerializer File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/write/dataframe_serializer.py", line 12, in from influxdb_client.client.write.point import _ESCAPE_KEY, _ESCAPE_STRING, _ESCAPE_MEASUREMENT, DEFAULT_WRITE_PRECISION File "/var/mycodo-root/env/lib/python3.9/site-packages/influxdb_client/client/write/point.py", line 40, in import numpy as np File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 135, in raise ImportError(msg) from e ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python interpreter from there.

it just loops the "The above exception was the direct cause of the following exception" basically for >10.000 lines of code

[kizniche]

Have you attempted to upgrade the library or install the dependency it lists?

[iqgmoritz]

Yes I've tried to install everything by hand even before installing mycodo on a new system. Still the same error

[iqgmoritz]

In the web log it's actually a bit better to understand: Nov 08 22:07:09 raspberrypi python[978]: ImportError: Nov 08 22:07:09 raspberrypi python[978]: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Nov 08 22:07:09 raspberrypi python[978]: Importing the numpy C-extensions failed. This error can happen for Nov 08 22:07:09 raspberrypi python[978]: many reasons, often due to issues with your setup or how NumPy was Nov 08 22:07:09 raspberrypi python[978]: installed. Nov 08 22:07:09 raspberrypi python[978]: We have compiled some common reasons and troubleshooting tips at: Nov 08 22:07:09 raspberrypi python[978]: https://numpy.org/devdocs/user/troubleshooting-importerror.html Nov 08 22:07:09 raspberrypi python[978]: Please note and check the following: Nov 08 22:07:09 raspberrypi python[978]: The Python version is: Python3.9 from "/var/mycodo-root/env/bin/python" Nov 08 22:07:09 raspberrypi python[978]: The NumPy version is: "1.26.1" Nov 08 22:07:09 raspberrypi python[978]: and make sure that they are the versions you expect. Nov 08 22:07:09 raspberrypi python[978]: Please carefully study the documentation linked above for further help. Nov 08 22:07:09 raspberrypi python[978]: Original error was: libopenblas.so.0: cannot open shared object file: No such file or directory Nov 08 22:07:09 raspberrypi python[978]: The above exception was the direct cause of the following exception: Nov 08 22:07:09 raspberrypi python[978]: Traceback (most recent call last): Nov 08 22:07:09 raspberrypi python[978]: File "/home/moritz/Mycodo/mycodo/devices/camera.py", line 372, in camera_record Nov 08 22:07:09 raspberrypi python[978]: import cv2 Nov 08 22:07:09 raspberrypi python[978]: File "/var/mycodo-root/env/lib/python3.9/site-packages/cv2/init.py", line 11, in Nov 08 22:07:09 raspberrypi python[978]: import numpy Nov 08 22:07:09 raspberrypi python[978]: File "/var/mycodo-root/env/lib/python3.9/site-packages/numpy/init.py", line 135, in Nov 08 22:07:09 raspberrypi python[978]: raise ImportError(msg) from e Nov 08 22:07:09 raspberrypi python[978]: ImportError: Error importing numpy: you should not try to import numpy from Nov 08 22:07:09 raspberrypi python[978]: its source directory; please exit the numpy source tree, and relaunch Nov 08 22:07:09 raspberrypi python[978]: your python interpreter from there. Nov 08 22:07:09 raspberrypi python[978]: 2023-11-08 22:07:09,473 Exception raised in 'camera_record' when setting user grp: [Errno 2] No such file or directory: '/home/moritz/Mycodo/cameras/6fd9fa62-5a7b-4491-b806-3827ad9e10c5/still/Still-2-Camera_Name-2023-11-08_22-07-06.jpg' Nov 08 22:07:09 raspberrypi python[978]: Traceback (most recent call last): Nov 08 22:07:09 raspberrypi python[978]: File "/home/moritz/Mycodo/mycodo/devices/camera.py", line 488, in camera_record Nov 08 22:07:09 raspberrypi python[978]: set_user_grp(path_file, 'mycodo', 'mycodo') Nov 08 22:07:09 raspberrypi python[978]: File "/home/moritz/Mycodo/mycodo/utils/system_pi.py", line 683, in set_user_grp Nov 08 22:07:09 raspberrypi python[978]: os.chown(filepath, uid, gid) Nov 08 22:07:09 raspberrypi python[978]: FileNotFoundError: [Errno 2] No such file or directory: '/home/moritz/Mycodo/cameras/6fd9fa62-5a7b-4491-b806-3827ad9e10c5/still/Still-2-Camera_Name-2023-11-08_22-07-06.jpg' Nov 08 22:07:09 raspberrypi python[978]: 2023-11-08 22:07:09,510 No image was created. Check your settings and hardware for any issues. Nov 08 22:07:09 raspberrypi python[978]: 2023-11-08 22:07:09,511 Could not acquire image.

[kizniche]

How did you install the dependency?

[iqgmoritz]

On the initial try I did it via the Mycodo web interface, then when it did not work I did a clean install of Raspbian Lite and installed numpy via $ pip install numpy

[kizniche]

I'm referring to the dependency numpy told you about in the log lines you pasted.

[iqgmoritz]

As I said, i installed it via the mycodo web interface prompt when i tried to add the urllib camera. I tried it again on a freshly installed Raspberry Pi by running the $ pip install numpy first and only after that installing mycodo on the system. Apparently in both cases, mycodo downloads it‘s own version of numpy to /var/mycodo-root/env/lib/python3.9/site-packages/numpy/

I‘ve also tried to somehow get rid of this installation but pip uninstall does not work and just deleting the directory and hoping for python to access the user installed numpy dependency did not either.

[kizniche]

How did you install libopenblas (the dependency numpy is stating you're missing in the log you pasted) from the Mycodo web interface? This is not a feature of Mycodo unless you modified the code.

[iqgmoritz]

I never willingly installed libatlas. Though I am running gphoto2 on the pi to take automated timelapse photos with a DSLR, which I wanted to be able to import to mycodo with the urllib camera, so this might be the issue. I will try it on a system without gohoto2 later today.

[kizniche]

Who mentioned libatlas?

[iqgmoritz]

Oh, sorry excuse the confusion. I misread the log and your message. I installed it via $ sudo apt-get install libopenblas-dev

[iqgmoritz]

I have now manually compiled openblas and the error message has shrunken significantly to this:

Nov 10 15:19:19 raspberrypi python[590]: Traceback (most recent call last): Nov 10 15:19:19 raspberrypi python[590]: File "/home/moritz/Mycodo/mycodo/devices/camera.py", line 488, in camera_record Nov 10 15:19:19 raspberrypi python[590]: set_user_grp(path_file, 'mycodo', 'mycodo') Nov 10 15:19:19 raspberrypi python[590]: File "/home/moritz/Mycodo/mycodo/utils/system_pi.py", line 683, in set_user_grp Nov 10 15:19:19 raspberrypi python[590]: os.chown(filepath, uid, gid) Nov 10 15:19:19 raspberrypi python[590]: FileNotFoundError: [Errno 2] No such file or directory: '/home/moritz/Mycodo/cameras/195f9671-eb1d-4a49-8f76-1106d50557eb/still/Still-1-Camera_Name-2023-11-10_15-19-16.jpg' Nov 10 15:19:19 raspberrypi python[590]: 2023-11-10 15:19:19,295 No image was created. Check your settings and hardware for any issues. Nov 10 15:19:19 raspberrypi python[590]: 2023-11-10 15:19:19,295 Could not acquire image.

I cannot make much sense of this, as it seems like there is already an error beforehand when creating the jpg file, which is just not logged

[iqgmoritz]

I finally got it working, so here are the steps I followed for anyone with the same problem:

1. I compiled openblas myself with the following commands: wget http://github.com/xianyi/OpenBLAS/archive/v0.2.20.tar.gz tar zxvf v0.2.20.tar.gz cd OpenBLAS make mkdir ../openblas-0.2.20 make PREFIX=/home/pi/openblas-0.2.20 install

2. I installed opencv myself with: sudo apt-get install build-essential cmake pkg-config libjpeg-dev libtiff5-dev libjasper-dev libpng-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libfontconfig1-dev libcairo2-dev libgdk-pixbuf2.0-dev libpango1.0-dev libgtk2.0-dev libgtk-3-dev libatlas-base-dev gfortran libhdf5-dev libhdf5-serial-dev libhdf5-103 python3-pyqt5 python3-dev -y pip install opencv-python

Now it works. Thanks for your help anyways, without you I wouldn't have noticed, where the core of the problem is coming from!

[kizniche]

Although this may work for a manual fix, this isn't a viable solution for a software fix. I will attempt to replicate the issue and find an amenable solution that can be applied.

[FlooSz]

My solution was with checking journal: sudo journalctl -u mycodo.service

Then

1. Go into virtualenv with: 1. "cd Mycodo" 2. "source env/bin/activate" (without ")

2. Check if your numpy installed correctly with: env/bin/pip install numpy

3. Install openblas with: sudo apt-get install libopenblas-base (And/or other Packages you found with first journalctl command)

[kizniche]

I couldn't replicate the issue. Perhaps your issue stems from your use of a non-current operating system version. In any case, I can't do any diagnostics if it can't be replicated and I will only use the latest OS version, since it's too much trouble to support old OS versions.

[iqgmoritz]

That‘s pretty weird. I got the Raspberry Pi OS (Legacy) Lite 32bit from the Rasperry Pi Flasher, so it should have been the newest

[kizniche]

By definition, legacy is not the latest

[FlooSz]

For optimal performance with Raspi 4b, it is recommended to utilize a 64-bit operating system to unlock its full processing capabilities.

[iqgmoritz]

I was only using the 32bit version, because when I installed mycodo, weirdly this version was the only one which I was even capable of installing it on. The other OS‘s always had some error show up along the way

[kizniche]

Legacy does not mean 32-bit, it means old version