Person detection accuracy with Edge TPU vs OpenCV?

[ibrewster]

I've been playing around with the core image recognition code, simply instantiating a model and calling model.detect(), using code like the following:

import cv2

import pyzm.ml.object as object_detection
import pyzm.ZMLog as log

log.init(name='zmdetect_test', override = {'log_level_debug': 2,
                                            'log_debug': 1,
                                            'log_debug_file': '/var/log/zoneminder/zmdetect_testd.log',})

config = {'object_framework': 'coral_edgetpu',
          'object_min_confidence': .3,
          'tpu_max_processes': 1,
          'object_weights': '/var/lib/zmeventnotification/models/coral_edgetpu/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite',
          'object_labels': '/var/lib/zmeventnotification/models/coral_edgetpu/coco_indexed.names'}

model = object_detection.Object(logger=log, options=config)
img=cv2.imread('/tmp/3_326968_alarm.jpg')
model.detect(img)

When I set the config to use yolov4, it detects a person in my test image with 87.6% confidence, as it should. However, when I run it as above, using the coral edge TPU, it detects nothing. Note that this is not always the case, it often detects something, however it seems much less likely to detect people than OpenCV does. Is this just a limitation of the model, or is there something I can do to improve this?

[ibrewster]

So in a bit more testing, using the TPU it does detect the person, but only with a .0664 (6.6%, vs 87.6% for OpenCV) confidence. This is the image I am using, if that helps any:

[pliablepixels]

TPU and OpenCV use different models. Not a bug, but a property of the model being used.

[neillbell]

I was initially excited about the TPU support for object detection because I have a pretty lousy GPU and the TPU would greatly cut down on model processing time. After seeing the same sort of performance as @ibrewster, however, I went back to Yolov4/GPU.

Do you have an opinion on the relatively poor performance of the TPU system? Is it anything inherent in the system, or is the model just not as mature as Yolo?

[ibrewster]

I did a bit more digging, and found this statement on https://coral.ai/models/ referring to the model that the install script downloads:

Notice: These are not production-quality models; they are for demonstration purposes only.

So I have to wonder if that is the whole issue? And if so, how might we go about creating/obtaining a production-quality model for the TPU?

[pliablepixels]

@neillbell and @ibrewster are talking about two different things as I see it.

@ibrewster is talking about accuracy. @neillbell is talking about performance.

First on performance: a) Make sure you are on USB 3.0 b) You can switch tpu library to the overclocked version - see coral.ai for more details and warnings c) I am not sure what performance you are seeing on your GPU, but in general, compare the TPU performance to published specs on the internet. If it is far off, we can talk about optimization. If it's close, that's the limit.

On accuracy: a) I've always found mobile ssdnet to be less accurate than yolo. Even on desktop. I don't quite know the impact of non production models. It may be mean they are not tuned optimally or maybe trained with lesser images. Don't know. My core focus here is to enable the platform. I'd suggest you continue your research on how to optimize it and potentially find more optimal models

[ibrewster]

For what it's worth, the performance I am seeing is excellent - roughly 100X better than my video card (around 7ms vs 700ms). So no concerns there. I was just assuming from the phrase "After seeing the same sort of performance as @ibrewster" that @neillbell was referring to the same issues I was, and just used the term "performance" rather than "accuracy" Of course, they can speak for themselves :-)

My core focus here is to enable the platform. I'd suggest you continue your research on how to optimize it and potentially find more optimal models

Understood

[neillbell]

No, I was also talking about accuracy. TPU performance has been fantastic. It can very quickly fail to find a large number of objects. :)

With GPU Yolov4 takes about 110-150ms per run. I only have a 3.0 compute capability, so I'm happy that it works at all. It's still faster than CPU. CPU can take as much as 600-800ms.

TPU runs around 12ms.

[neillbell]

I hope we are not sounding ungrateful. The fact that we have all the processor and model choices is nothing short of miraculous. You have done an amazing job.

[ibrewster]

I hope we are not sounding ungrateful. The fact that we have all the processor and model choices is nothing short of miraculous. You have done an amazing job.

Indeed. Any weakness in the model is certainly not the fault of anyone here. Now if only I knew how to make a better model, or better yet, convert the yolov3/4 model to work... :-)

[pliablepixels]

No, I was also talking about accuracy. TPU performance has been fantastic. It can very quickly fail to find a large number of objects. :)

Ah ok, I thought you meant performance because I read:

TPU would greatly cut down on model processing time.

I hope we are not sounding ungrateful.

Oh goodness no. In reviewing my response, it may have sounded curt and that was likely because I was multi-tasking. By no means did I consider it to be a slight. I did mean it seriously though (and without intending to sound irritated) - I think there are areas of improvement but I just don't have the time to investigate. That being said, I'd love for someone to get a working yolov4 or a "production" ssdnet model (whatever that means) working and post here

Now if only I knew how to make a better model, or better yet, convert the yolov3/4 model to work... :-

For both, there are actually many internet resources. This is for tinyYOLOv4 https://github.com/guichristmann/edge-tpu-tiny-yolo This is for Yolov4 https://github.com/hunglc007/tensorflow-yolov4-tflite but it did not work for me (and I did not experiment more)

How to make a better model: You train it and tune it. https://coral.ai/docs/edgetpu/retrain-detection/#requirements for retraining, but the tools are the same (using edgetpu compiler). https://github.com/f0cal/google-coral is a great place to ask questions.

[ibrewster]

For both, there are actually many internet resources. This is for tinyYOLOv4 https://github.com/guichristmann/edge-tpu-tiny-yolo This is for Yolov4 https://github.com/hunglc007/tensorflow-yolov4-tflite but it did not work for me (and I did not experiment more)

Yeah. So far, no luck. I get as far as the Edge TPU Computer command, but that always errors out with a "model not quantized" error, even though I've followed the steps to quantize the model to a T, and they complete apparently without error. There might be a few more things I can try still though. Thanks for the pointers!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.