Camera resource not released properly

[sajmons]

I'm having problem with indi_pylibcamera driver after it was not properly disconected. It looks like that some process is still using camera. I tried to kill indiserver, restart indi_web_manager. Nothing helps. Only reboot or kill process python3 solves the problem.

Here is output of libcamera-hello after indi_pylibcamera not disconected properly.

simon@raspberrypi:~ $ killall -9 indiserver
simon@raspberrypi:~ $ libcamera-hello -t0
Made X/EGL preview window
[23:31:26.386909776] [178147]  INFO Camera camera_manager.cpp:299 libcamera v0.0.4+22-923f5d70
[23:31:26.460912891] [178148]  INFO RPI raspberrypi.cpp:1476 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media4 and ISP device /dev/media0
[23:31:26.461703565] [178147]  INFO Camera camera.cpp:841 Pipeline handler in use by another process
ERROR: *** failed to acquire camera /base/soc/i2c0mux/i2c@1/imx477@1a ***
simon@raspberrypi:~ $ libcamera-hello -t0
Made X/EGL preview window
[23:35:40.759566079] [178679]  INFO Camera camera_manager.cpp:299 libcamera v0.0.4+22-923f5d70
[23:35:40.807548553] [178683]  INFO RPI raspberrypi.cpp:1476 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media4 and ISP device /dev/media0
[23:35:40.807800031] [178679]  INFO Camera camera.cpp:841 Pipeline handler in use by another process
ERROR: *** failed to acquire camera /base/soc/i2c0mux/i2c@1/imx477@1a ***
simon@raspberrypi:~ $ lsof /dev/video0
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python3 173840 simon   15u   CHR  81,14      0t0  483 /dev/video0
simon@raspberrypi:~ $ lsof /dev/media0
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python3 173840 simon   12uW  CHR  238,0      0t0  221 /dev/media0
simon@raspberrypi:~ $ 

It looks like python3 process is still using camera after drver crashed.

After I kill that process manually everithing is working again. Ekos connect properly.

simon@raspberrypi:~ $ kill -9 173840
simon@raspberrypi:~ $ libcamera-hello -t0
Made X/EGL preview window
[23:43:14.416553017] [179629]  INFO Camera camera_manager.cpp:299 libcamera v0.0.4+22-923f5d70
[23:43:14.477593590] [179630]  INFO RPI raspberrypi.cpp:1476 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media4 and ISP device /dev/media0
[23:43:14.478633428] [179629]  INFO Camera camera.cpp:1028 configuring streams: (0) 2028x1520-YUV420
[23:43:14.479021997] [179630]  INFO RPI raspberrypi.cpp:851 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 2028x1520-SBGGR12_1X12 - Selected unicam format: 2028x1520-pBCC
simon@raspberrypi:~ $ 

Is it possible to somehow automate this step? Maybe you could remember that python3 processid on driver startup and write it down to some file. On next startup you check that file and if it containt processid you kill it?

[scriptorron]

This is likely the effect I described in the README.md in section "Error when restarting indiserver". The INDI server starts the driver as a sub-process and when the server crashes the sub-process does not get terminated.

There can only be one indi_pylibcamera.py driver running on the machine. I will implement something that kills an existing driver at startup. I just need to find out how to do this in Python (without killing itself). That will certainly not work when the older driver was started by a different user, but that is not a normal use-case.

[sajmons]

That would be great! Indi web manager automatically restart driver when necessary. It would greatly improved UX and robustness if you can figure that out.

[scriptorron]

I implemented the killing of an old driver in branch https://github.com/scriptorron/indi_pylibcamera/tree/35-camera-resource-not-released-properly It worked in my tests. But I found another issue: when you disconnect a camera and connect it again the following error comes up:

RuntimeError: Camera __init__ sequence did not complete.

It is reproducible by clicking in the driver "Connect", "Disconnect" and "Connect". This can also happen when you kill a driver which had a camera connected, start a new driver and connect the camera in the new driver. After a while it works. It seems, the libcamera stack still has a resource locked until a timeout occurs or a processing finished. I will try to find a solution.

[scriptorron]

I fixed the reconnect issue in the mentioned branch. It was easier than expected. Next days I will do more tests before I merge it to the master branch.

[sajmons]

That's really great to hear that you managed to solve that problem. It should really graatly improve stability and robustness of imaging. Thumbs up!

[scriptorron]

Today I also added code to close camera when getting killed by SIGINT or SIGTERM. I will merge to main branch. Maybe you can give ma feedback if it works in your environment. Thanks.

[sajmons]

Wow! Latest version feels amazing! Changing different INDI clients feels great. There is no more issuse if user don't disconect properly. Driver just works! You did amazing job! Driver is now really robust!

[scriptorron]

Thank you for your kind words. You have a big part on it: you test it in an environment I do not have. Thank you for your help!

[sajmons]

I'm really glad that I can help.