Random Crash #2

[michi1996]

Hi, Hatte nun auch schon mehrere male den Absturz mit diesem Fehler, evtl. kannst du mit dem Output etwas anfangen:

No Face Found...
Total Runtime: 1.354121446609497
Runtime: 1.354191541671753
Timestamp at Done Runtime: 2020_10_21_05-33-24.251932
Overhead: 24.625928
CUMULUS: 0
Working the Queque with len: 37
2020_10_21_05-33-00.013303
Traceback (most recent call last):
  File "cascade.py", line 727, in <module>
    sq_cascade.queque_handler()
  File "cascade.py", line 336, in queque_handler
    self.queque_worker()
  File "cascade.py", line 232, in queque_worker
    cascade_obj = self.feed(target_img=self.main_deque[self.fps_offset][1], img_name=self.main_deque[self.fps_offset][0])[1]
  File "cascade.py", line 383, in feed
    single_cascade = self.base_cascade.do_single_cascade(event_img_object=target_event_obj)
  File "cascade.py", line 413, in do_single_cascade
    dk_bool, cat_bool, bbs_target_img, pred_cc_bb_full, cc_inference_time = self.do_cc_mobile_stage(cc_target_img=cc_target_img)
  File "cascade.py", line 561, in do_cc_mobile_stage
    pred_cc_bb_full, cat_bool, inference_time = self.cc_mobile_stage.do_cc(target_img=cc_target_img)
  File "/home/pi/CatPreyAnalyzer/model_stages.py", line 98, in do_cc
    preprocessed_img = self.resize_img(input_img=target_img)
  File "/home/pi/CatPreyAnalyzer/model_stages.py", line 93, in resize_img
    img = cv2.cvtColor(input_img, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.4.0) /tmp/pip-install-aty2tyyk/opencv-python/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 15116544 bytes in function 'OutOfMemoryError'

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pi/CatPreyAnalyzer/camera_class.py", line 38, in fill_queue
    image = cv2.imdecode(data, 1)
cv2.error: OpenCV(4.4.0) /tmp/pip-install-aty2tyyk/opencv-python/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 15116544 bytes in function 'OutOfMemoryError'

Beste Grüsse Michi

[niciBume]

Danke für den print! Scheint mir als würde ich mein memory nicht richtig clear'en, ich schaue mal rein 👍

[benleb]

I think the message refers to the memory of the GPU, isn't it? Maybe there is too less available at all? i saw similar errors on a jetson nano when using opencv...

[niciBume]

I'm not quite sure if OpenCV default goes to the GPU. I haven't specifically implemented it to do so. Either way, I gave it a few tweaks. Lowered the max Queue Threshold, clear the camera and force garbage collection. Use the newest code, it should also stabilise inference time of the networks by running a "Calibration image" in the beginning. Hope this helps mitigate the error. It's a bit tricky as the error isn'r really reproducible...

Don't forget to save the telegram data somewhere before pulling!

[michi1996]

I've just loaded the new code and started the program. I will watch this cautiously and give feedback here if the problem occurs again.

Thank you very much for the quick fix.

[niciBume]

No worries! Hope this works, very annoying bug. Cheers

[michi1996]

Unfortunately the same crash again today

NO CAT FOUND!
Working the Queque with len: 27
2020_10_23_07-38-02.321458
Traceback (most recent call last):
  File "cascade.py", line 718, in <module>
    sq_cascade.queque_handler()
  File "cascade.py", line 310, in queque_handler
Quelength: 28   Streamsize: 48
    self.queque_worker()
  File "cascade.py", line 203, in queque_worker
    cascade_obj = self.feed(target_img=self.main_deque[self.fps_offset][1], img_name=self.main_deque[self.fps_offset][0])[1]
  File "cascade.py", line 359, in feed
    single_cascade = self.base_cascade.do_single_cascade(event_img_object=target_event_obj)
  File "cascade.py", line 418, in do_single_cascade
    dk_bool, cat_bool, bbs_target_img, pred_cc_bb_full, cc_inference_time = self.do_cc_mobile_stage(cc_target_img=cc_target_img)
  File "cascade.py", line 566, in do_cc_mobile_stage
    pred_cc_bb_full, cat_bool, inference_time = self.cc_mobile_stage.do_cc(target_img=cc_target_img)
  File "/home/pi/CatPreyAnalyzer/model_stages.py", line 98, in do_cc
    preprocessed_img = self.resize_img(input_img=target_img)
  File "/home/pi/CatPreyAnalyzer/model_stages.py", line 93, in resize_img
    img = cv2.cvtColor(input_img, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.4.0) /tmp/pip-install-aty2tyyk/opencv-python/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 15116544 bytes in function 'OutOfMemoryError'

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pi/CatPreyAnalyzer/camera_class.py", line 39, in fill_queue
    image = cv2.imdecode(data, 1)
cv2.error: OpenCV(4.4.0) /tmp/pip-install-aty2tyyk/opencv-python/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 15116544 bytes in function 'OutOfMemoryError'

[niciBume]

Testing new stuff on my node. I'll update the code as soon as my node survives 48 hours.

[niciBume]

Two nodes have been running continuously for ~48hours without crash. Problem seems to have been the wrong releasing of memory. Because RAM increased steadily with the old code. Now the memory has stabilised at ~900M on the RPI4 4GB model. Hope it runs smoothly now 👍

[niciBume]

Should be fixed now