[BUG] pose_engine.py runtime exception

[ivelin]

Describe the bug Pose engine throws intermittent runtime exceptions.

To Reproduce Run pose engine through a variety of fall images.

Expected behavior There should be no unexpected runtime exceptions caused by image processing during fall detections.

Host environment (please complete the following information):

• Hardware: Raspberry Pi
• OS: Raspberry OS
• Container: docker
• Version: ambianic edge 1.13.2

Additional context

2021-02-07 13:09:02 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1077.19 ms, 0.93 fps in pipeline area_watch
2021-02-07 13:09:02 ERROR /opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py.process_sample(101): Error "only size-1 arrays can be converted to Python scalars" while processing sample. Dropping sample: {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1280x720 at 0xADB66570>}
Traceback (most recent call last):
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py", line 85, in process_sample
    inference_result, thumbnail = self.fall_detect(image=image)
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py", line 344, in fall_detect
    pose, thumbnail, pose_score, pose_dix = self.find_keypoints(image)
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py", line 146, in find_keypoints
    poses, thumbnail = self._pose_engine.DetectPosesInImage(image)
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/pose_engine.py", line 146, in DetectPosesInImage
    kps = self.parse_output(template_heatmaps, template_offsets, 0.3)
  File "/opt/ambianic-edge/src/ambianic/pipeline/ai/pose_engine.py", line 89, in parse_output
    pose_kps[i, 0] = int(remap_pos[0] + offset_data[max_val_pos[0], max_val_pos[1], i])
TypeError: only size-1 arrays can be converted to Python scalars
2021-02-07 13:09:03 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1070.10 ms, 0.70 fps in pipeline area_watch
2021-02-07 13:09:04 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1003.87 ms, 0.99 fps in pipeline area_watch
2021-02-07 13:09:05 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1042.34 ms, 0.96 fps in pipeline area_watch

[bhavikapanara]

@ivelin

Can you please provide me with a sample test fall image so I can reproduce the error and fix it?

[ivelin]

No, because the error is intermittent and since it is not handled in a way that provides much context, I cannot correlate the message to an image. Please think how we can better encapsulate this error with a contextual information in the log that allows us to track it down.

[bhavikapanara]

ok...will check it.

[bhavikapanara]

@ivelin

I have tried to trace this issue but unable to locate the bug. This function use output data(heatmap_data, offset_data) that are generated from TFLite posenet model.

pose_kps[i, 0] = int(remap_pos[0] + offset_data[max_val_pos[0], max_val_pos[1], i])

pose_kps[i, 0] = int(remap_pos[0] + offset_data[max_val_pos[0], max_val_pos[1], i])
TypeError: only size-1 arrays can be converted to Python scalars

And this error is raised when the function expects a single value but you pass an array instead.

I am trying to check at what circumstance model output is different. But not able to find it. Can you please help me with your thoughts to deal with this bug?

thanks

[ivelin]

It's not obvious to me what the issue is. Is it possible that remap_pos[0] returns anything but a scalar? Same question for offset_data[max_val_pos[0], max_val_pos[1], i].

One of these two returns a list or array, which confuses int().

I suggest surrounding these lines with an exception handling block that provides as much context as needed. That way the next time this happens, we would know exactly what caused it.