indi_pylibcamera crashes and doesn't terminate occasionally

[FireNWater]

I am running kstars/ekos on a Raspberry Pi-5 w/NVME drive and AstroArch operating system. The developers of AA are using indipylibcamera to drive the Raspberry Pi HQ camera. This camera previously worked with Stellarmate and RpiOS. . When not_ starting from a fresh boot, we would get crashes from the indi_pylibcamera driver while starting the INDI drivers from ekos. This is the error message from the log file:

[2024-09-14T14:20:23.713 UTC INFO ][           org.kde.kstars.indi] - Updating "indi_pylibcamera" Location Longitude: 269.XXX Latitude: 35.XXXX Elevation: 82.9
[2024-09-14T14:20:23.714 UTC INFO ][           org.kde.kstars.ekos] - Ekos received a new device:  "indi_pylibcamera"
[2024-09-14T14:20:23.714 UTC INFO ][           org.kde.kstars.indi] - Updating "indi_pylibcamera" Location Longitude: 269.XXX Latitude: 35.XXXX Elevation: 82.9
[2024-09-14T14:20:23.977 UTC INFO ][           org.kde.kstars.ekos] - Connecting to "indi_pylibcamera"
[2024-09-14T14:20:24.029 UTC INFO ][           org.kde.kstars.indi] - indi_pylibcamera :  "[INFO] connecting to camera /base/axi/pcie@120000/rp1/i2c@88000/imx477@1a "
[2024-09-14T14:20:24.029 UTC INFO ][           org.kde.kstars.indi] - indi_pylibcamera :  "[INFO] closing camera "
[2024-09-14T14:20:24.029 UTC INFO ][           org.kde.kstars.indi] - indi_pylibcamera :  "[INFO] opening camera "
[2024-09-14T14:20:24.093 UTC INFO ][           org.kde.kstars.indi] - indi_pylibcamera :  "[ERROR] Uncaught exception!&#10;Traceback (most recent call last):&#10;  File \"/usr/lib/python3.12/site-packages/picamera2/picamera2.py\", line 254, in __init__&#10;    self._open_camera()&#10;  File \"/usr/lib/python3.12/site-packages/picamera2/picamera2.py\", line 465, in _open_camera&#10;    self.camera.acquire()&#10;RuntimeError: Failed to acquire camera: Device or resource busy&#10;&#10;During handling of the above exception, another exception occurred:&#10;&#10;Traceback (most recent call last):&#10;  File \"/usr/bin/indi_pylibcamera\", line 33, in <module>&#10;    sys.exit(load_entry_point('indi-pylibcamera==2.6.4', 'console_scripts', 'indi_pylibcamera')())&#10;             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&#10;  File \"/usr/lib/python3.12/site-packages/indi_pylibcamera/indi_pylibcamera.py\", line 1372, in main&#10;    device.run()&#10;  File \"/usr/lib/python3.12/site-packages/indi_pylibcamera/indidevice.py\", line 898, in run&#10;    self.message_loop()&#10;  File \"/usr/lib/python3.12/site-packages/indi_pylibcamera/indidevice.py\", line 844, in message_loop&#10;    vector.set_byClient(values)&#10;  File \"/usr/lib/python3.12/site-packages/indi_pylibcamera/indi_pylibcamera.py\", line 126, in set_byClient&#10;    if self.parent.openCamera():&#10;       ^^^^^^^^^^^^^^^^^^^^^^^^&#10;  File \"/usr/lib/python3.12/site-packages/indi_pylibcamera/indi_pylibcamera.py\", line 740, in openCamera&#10;    self.CameraThread.openCamera(CameraIdx)&#10;  File \"/usr/lib/python3.12/site-packages/indi_pylibcamera/CameraControl.py\", line 361, in openCamera&#10;    self.picam2 = Picamera2(idx)&#10;                  ^^^^^^^^^^^^^^&#10;  File \"/usr/lib/python3.12/site-packages/picamera2/picamera2.py\", line 266, in __init__&#10;    raise RuntimeError(\"Camera __init__ sequence did not complete.\")&#10;RuntimeError: Camera __init__ sequence did not complete. "
[2024-09-14T14:20:24.337 UTC CRIT ][           org.kde.kstars.indi] - INDI driver  "indi_pylibcamera"  crashed!
[2024-09-14T14:20:24.341 UTC INFO ][           org.kde.kstars.ekos] - "indi_pylibcamera is offline."

 After checking "htop" from the linux command line, it showed several "indi_pylibcamera" processes still running from the previous run and INDI shutdown.  If I then killed those processes, the inid_pylibcamera driver would start normally from ekos.

 It appears there are two main issues going on.

1) The indi_pylibcamera driver is not properly terminating every time that ekos shuts down the INDI drivers.
2) The indi_pylibcamera driver is not tolerant of a previously running instance of itself.

Hope this helps. Thank you for making this driver!!

[scriptorron]

The error message 'Failed to acquire camera: Device or resource busy' indicates that the camera is in use by an other software. You already found that there are more than one instance of 'indi_pylibcamera' running at the same time.

I suppose you have a recent version (>2.1) of the 'indi_pylibcamera' driver. Since that version the driver detects other instances of itself and kills them. For detection it runs 'ps -ax' and searches 'python3' and 'indi_pylibcamera' in the lines. Could you please check:

1. Is 'ps -ax' available on your system?
2. In the error situation (previous driver instance still running): please run 'ps -ax' and forward me the output.

Maybe AstroArch has changed the driver name or it starts the driver with 'python' and not with 'python3'. We will see that in the 'ps -ax' output.

[FireNWater]

Thank you for your reply. Yes, "ps-ax" is available on AstroArch. I will try to re-create the issue today now that I know where to look. I'm running the driver on an Rpi-5, so ekos is the only app using the driver. I will notify Matt of your reply, this seems like it should fix the problem.

[scriptorron]

I can also improve the detection of a running instance. But first I would need to know why it is not detected. The ps -ax output would really help.

[FireNWater]

I have a test Rpi, I will try to recreate and get you the output this afternoon.

[FireNWater]

pylibcameraerror.txt . That's the output from the ps -ax command when the pylibcamera driver continues running . .

[scriptorron]

I see. In your AstroArch the driver gets started with python and not with python3:

11044 ?        Sl     1:07 /usr/bin/python /usr/bin/indi_pylibcamera

That's the reason why a running instance does not get detected when restarting the driver.

During the transition from Python 2 to Python 3 it was almost standard to start Python 2 with python and Python 3 with python3. Nowadays Python 2 disappears from distributions and it seems that python becomes the new command to start Python 3.

Unfortunately the indi_pylibcamera driver looks for python3. I will fix this in the next version. For the meantime you can:

1. search for the file indi_pylibcamera.py (it is likely somewhere in /lib/python*/ and you can search it with the command find /lib/python* -name indi_pylibcamera.py)
2. open indi_pylibcamera.py with a text editor (for instance nano) and edit line 563: original: modified: replace python3 by python
3. save the modified file.

That should fix the issue for the current version.

[FireNWater]

Perfect, thank you so much for your quick response. I will forward this to Matt.

Would it be better to ask Matt to use python3 in future instead of python?