themarpe
commented
1 year ago Hi @dhartness
Thanks for the report - I have some suspicions as to what might cause this, but nothing stands out as "the obvious" thing.
The issue stems from the native library and is therefore "uncatchable" from Python side.
I think it'd be best if you set the creating of coredump files & share that after this crash.
echo 'core' | sudo tee /proc/sys/kernel/core_pattern
ulimit -c unlimited
# Then execute the scriptIf you see the above crash you should get a core.[id] file - we'll be able to analyze that, to see into what state the system got for it to crash in such manner.
dhartness
diablodale
After running for several days in a script that utilizes six OAK POE devices, five OAK-1 and one OAK-D, one of them threw the following error and crashed the entire script:
` Stack trace (most recent call last) in thread 8936:
6 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
5 Object "/lib/aarch64-linux-gnu/libc.so.6", at 0x7f92638c1b, in
4 Object "/lib/aarch64-linux-gnu/libpthread.so.0", at 0x7f92817647, in
3 Object "/usr/local/lib/python3.9/dist-packages/depthai.cpython-39-aarch64-linux-gnu.so", at 0x7f887e104b, in
2 Object "/usr/local/lib/python3.9/dist-packages/depthai.cpython-39-aarch64-linux-gnu.so", at 0x7f886c587b, in
1 Object "/usr/local/lib/python3.9/dist-packages/depthai.cpython-39-aarch64-linux-gnu.so", at 0x7f887282c7, in dai::StreamMessageParser::parseMessageToADatatype(streamPacketDesc_t*)
0 Object "/lib/aarch64-linux-gnu/libc.so.6", at 0x7f925eb64c, in
Segmentation fault (Invalid permissions for mapped object [0x7f7489e000]) Segmentation fault hviz@hvizpi18:~/HVision $ sudo screendump 1 > term001.log`
At this point it has happened twice. We've normally observed the system, with a mix of up to ten cameras, running for over a week. Because of the infrequency we are unable to reproduce and it doesn't indicate a line number so we're unable to verify its not happening outside our try/except block.
I don't see any thrown exceptions on screen from the time the script was running and none were caught in the exception block.
Even if a camera has crashed, normally we catch it in an exception block and then reload the camera, but this bug has killed the entire script, dumping us back to the CLI, and I'm not sure how to determine which camera has caused it.
dthompson_mre_03202023.zip log_system_information.zip
This is the run() loop the camera uses: `while not self.stopped: singletimecall = time.perf_counter() if((singletimecall - checkforvaluechange) > 14): if self.rm.getmyneedtoreload(self.camindex): checkforvaluechange = singletimecall self.reloadcam() #############################
###################### self.trackletsData = None try: while self.needtoreinitialize: self.reloadcam() ################################################### imgFrame = self.preview.get() inDet = self.qDet.get() if imgFrame is not None: frame = imgFrame.getCvFrame() if(len(frametimestamps) > 1): dasfps = round((len(frametimestamps)/(frametimestamps[-1]-frametimestamps[0])),1) if (self.icountitems == 2) and (track is not None): self.trackletsData = track.tracklets if inDet is not None: detections = inDet.detections self.crop_queue.append((detections, frame, self.trackletsData, dasfps)) frametimestamps.append(time.perf_counter()) ################################################### self.recoveryattempts = 0 self.myhealth[self.camindex-1] = True self.myhealth[self.camindex-1] = self.healthychild[0] except Exception as exception: self.myhealth[self.camindex-1] = False exc_type, exc_obj, exc_tb = sys.exc_info() self.oaklogging.error(""+self.camlogident+"An error has occurred around the Oak Device while active: "+str(exc_type)+" "+str(exc_tb.tb_lineno)+"") self.oaklogging.error(""+self.camlogident+"Exception: {}".format(type(exception).name)+"") self.oaklogging.error(""+self.camlogident+"Exception message: {}".format(exception)+"") self.device.close() self.recoveryattempts += 1 self.needtoreinitialize = True`