search

cortictechnology / cep

CEP is a software platform designed for users that want to learn or rapidly prototype using standard A.I. components.
MIT License
166 stars 21 forks source link

RPi CSI Camera Configuration Issue #26

Closed topi-dev closed 2 years ago

topi-dev commented 2 years ago

First of all thank you for the great work you did implementing the CEP solution. After some tinkering I got it to work with an RPi /w CSI camera and a Lego Inventor Robot Kit. Your solution really is as much fun as it is educative and the interface is simply great.

However, after some tinkering I always arrive at a dead-end when initialization of the CSI camera will fail in during the setup of my program with the message "Initialization of vision failed. No CSI-Camera is detected, or connected device is not supported". At this point the camera preview on the setup screen also stops working. Camera and RPi are in a fixed setup, so there should not be any physical connection instabilities and work perfectly fine after the initial installation of CEP.

Rebooting the system or powering it off does not help. Once the issue happens I cannot recover the connection to the camera. I do believe the problem starts as soon as the setup phase of the program does fail once for any reason (e.g. the Lego Hub not being connected), but I am not certain.

Is there any solution to this problem or a feasible workaround? Can the camera setup be reset somehow?

I did check the camera connection using the Raspberry OS CommandLine-Tools and still-image as well as video work just fine. I did install the solution on the latest Raspberry OS Buster as CSI Camera Legacy mode in the most recent version caused issues.

Best regards, Tobias

ErrorMessage s

michaelhwn commented 2 years ago

Hi @topi-dev, I am very happy to hear that you like the project. Regarding the initialization failure, I have noticed it before on one Pi. When this happens, the dropdown menu of the "initialize vision with" block usually shows a "loading..." when you actually press and open it, instead of the actual "CSI-camera" device. I found that, if I close the dropdown and wait for a bit to open it again until it shows the "CSI-camera" device, initialization would succeed. Can you please confirm if you observe the same behaviour? If not, please execute the following commands during a fail initialization process and post the message here: sudo journalctl -u cait_webapp.service -f sudo journalctl -u curt_containers.service -f

topi-dev commented 2 years ago

Hi @michaelhwn and thank you for looking into this. With me the 'Initialize vision with' block correctly shows 'CSI Camera' and it is also the only choice in the dropdown. Same for the Setup-Screen. Reselecting the 'CSI Camera' entry does not resolve the issue. I attached the logs at the bottom,

InitializeVisionSelection

Log cait_webapp.service

Mar 14 20:17:21 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:21] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:21 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:21] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:21 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:21] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:21 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:21] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:21 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:21] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:21 robot1 bash[383]: WARNING:root:-------------------Nearby devices: [('64:FF:0A:7D:35:C5', 'SONY KD-55XH9005')]
Mar 14 20:17:24 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:24] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:24 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:24] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:24 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:24] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:24 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:24] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:24 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:24] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:24 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:24] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:26] "POST /get_cloud_accounts HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:26] "POST /get_cloud_accounts HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:26] "POST /get_nlp_models HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:26] "POST /get_nlp_models HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:26] "GET /getvideodev HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:26] "GET /getvideodev HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:26] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:26] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:26] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:26 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:26] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:27 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:27] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:27 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:27] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: WARNING:root:Vision User: pi
Mar 14 20:17:29 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /initialize_component HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /initialize_component HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /saveworkspace HTTP/1.1" 200 -
Mar 14 20:17:29 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:29] "POST /saveworkspace HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:32] "POST /get_cloud_accounts HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:32] "POST /get_cloud_accounts HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:32] "POST /get_nlp_models HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:32] "POST /get_nlp_models HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:32] "GET /getvideodev HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:32] "GET /getvideodev HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:32] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:32] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:32] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:32] "GET /getaudiodev HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:32] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:32 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:32] "POST /get_control_devices HTTP/1.1" 200 -
Mar 14 20:17:35 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:35] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:35 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:35] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:35 robot1 bash[383]: 192.168.2.130 - - [14/Mar/2022 20:17:35] "POST /get_states HTTP/1.1" 200 -
Mar 14 20:17:35 robot1 bash[383]: INFO:werkzeug:192.168.2.130 - - [14/Mar/2022 20:17:35] "POST /get_states HTTP/1.1" 200 -

Log curt_containers.service

Mar 14 20:18:35 robot1 bash[408]: curt_vision_1         |     self.load_configuration_file()
Mar 14 20:18:35 robot1 bash[408]: curt_vision_1         |   File "/data/curt/base_factory.py", line 54, in load_configuration_file
Mar 14 20:18:35 robot1 bash[408]: curt_vision_1         |     mod_class = getattr(mod, module_config[module])()
Mar 14 20:18:35 robot1 bash[408]: curt_vision_1         |   File "/data/curt/modules/vision/face_detection.py", line 23, in __init__
Mar 14 20:18:35 robot1 bash[408]: curt_vision_1         |     super().__init__("cpu",
Mar 14 20:18:35 robot1 bash[408]: curt_vision_1         |   File "/data/curt/modules/vision/tvm_processing.py", line 60, in __init__
Mar 14 20:18:35 robot1 bash[408]: curt_vision_1         |     os.rmdir(os.path.dirname(os.path.realpath(__file__))
Mar 14 20:18:35 robot1 bash[408]: curt_vision_1         | OSError: [Errno 39] Directory not empty: '/data/curt/modules/vision/../../../models/modules/vision/platforms/rpi32/tuned_blazeface_lib'
Mar 14 20:18:36 robot1 bash[408]: curt_curt_vision_1 exited with code 1
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | WARNING:root:Loading model lib now
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Exception ignored in: <function TempDirectory.__del__ at 0xa70cb8e0>
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Traceback (most recent call last):
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/root/tvm/python/tvm/contrib/utils.py", line 128, in __del__
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     self.remove()
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/root/tvm/python/tvm/contrib/utils.py", line 116, in remove
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     if self.temp_dir:
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | AttributeError: 'TempDirectory' object has no attribute 'temp_dir'
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | WARNING:root:Temp dir already exist, removing existing temp dir
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_rgb_camera_input
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_stereo_camera_input
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_spatial_calculator
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_face_detection
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_face_landmarks
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_face_recognition
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_object_detection
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_face_emotions
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_facemesh
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_pose_estimation
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_hand_landmarks
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Associating oakd pipeline to: oakd_hand_asl
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | Traceback (most recent call last):
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/modules/vision/main.py", line 17, in <module>
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     vision_main = VisionMain()
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/modules/vision/main.py", line 13, in __init__
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     super().__init__(VisionModule(), port=9436)
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/modules/vision_module.py", line 14, in __init__
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     super().__init__("vision")
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/base_module.py", line 49, in __init__
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     self.create_factory()
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/modules/vision_module.py", line 18, in create_factory
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     self.worker_factory = VisionFactory()
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/modules/vision/vision_factory.py", line 12, in __init__
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     super().__init__("vision")
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/base_factory.py", line 23, in __init__
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     self.load_configuration_file()
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/base_factory.py", line 54, in load_configuration_file
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     mod_class = getattr(mod, module_config[module])()
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/modules/vision/face_detection.py", line 23, in __init__
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     super().__init__("cpu",
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |   File "/data/curt/modules/vision/tvm_processing.py", line 60, in __init__
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         |     os.rmdir(os.path.dirname(os.path.realpath(__file__))
Mar 14 20:19:38 robot1 bash[408]: curt_vision_1         | OSError: [Errno 39] Directory not empty: '/data/curt/modules/vision/../../../models/modules/vision/platforms/rpi32/tuned_blazeface_lib'
Mar 14 20:19:40 robot1 bash[408]: curt_curt_vision_1 exited with code 1

It would be nice to keep the CSI camera. It's readily available and makes for a nice and clean setup. :-)

RobotSetup

Thank you, Tobias

michaelhwn commented 2 years ago

Thanks for the log @topi-dev, I see what happened now. There is a fix in the dev branch for the exact problem you are encountering, you can check out the dev branch and perform the setup on a newly flashed Raspberry Pi OS. To check out the dev branch, please perform the following steps:

git clone https://github.com/cortictechnology/cep.git
cd cep
git checkout dev

After setting up with the dev branch, the issue should be gone. Just let me know if the problem still persists.

michaelhwn commented 2 years ago

BTW, I like your set up a lot, I may want to set up one like that for myself too.

topi-dev commented 2 years ago

Thank you @michaelhwn for the info. I reinstalled the system using the current Dev-Branch and it is a lot more stable. It will even recover after a short while if I deliberately try to 'break' the setup action by adding a Lego Hub that is offline.

michaelhwn commented 2 years ago

I am glad to know it works for you.