cpu vs edgetpu detection speed

[balonchiks]

Hi,

I have trained my own tflite model based on ssd_mobilenet_v2 and it's working fine, however, i am barely seeing any improvement in detection speed when using edgetpu version and the usual one.

the model was that i am using for edgetpu is exactly the same model but converted to be used with official edgetpu_compiler.

while using those 2 models with a webcam script that i have on the same pi, i am seeing a drastic difference in persormance (1-2fps on cpu VS ~25fps on edgetpu) but this is what i get in doods apparently.

first log is cpu version:

2020-11-19T16:05:28.122Z    INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "default", "type": "tflite", "model": "models/jb_vb.tflite", "labels": 2, "width": 300, "height": 300}

2020-11-19T16:05:30.953Z    INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "default_edgetpu", "type": "tflite-edgetpu", "model": "models/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite", "labels": 80, "width": 300, "height": 300}

2020-11-19T16:05:32.410Z    INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "tensorflow", "type": "tensorflow", "model": "models/faster_rcnn_inception_v2_coco_2018_01_28.pb", "labels": 65, "width": -1, "height": -1}

2020-11-19T16:05:32.413Z    INFO    server/server.go:284    API Listening   {"package": "server", "address": ":8080", "tls": false, "version": "v0.2.5-0-gbf6d7a1-dirty"}

2020-11-19T16:05:33.646Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.134194637, "detections": 5, "device": null}

2020-11-19T16:05:33.648Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 1.109035679, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/uh0x0gFssG-000001", "remote": "10.0.10.20:58828"}

2020-11-19T16:05:43.595Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.122232716, "detections": 5, "device": null}

2020-11-19T16:05:43.596Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 1.090841654, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/uh0x0gFssG-000002", "remote": "10.0.10.20:46924"}

2020-11-19T16:05:53.568Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.122930258, "detections": 5, "device": null}

2020-11-19T16:05:53.571Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 1.101649318, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/uh0x0gFssG-000003", "remote": "10.0.10.20:58892"}

2020-11-19T16:06:03.796Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.147751694, "detections": 5, "device": null}

2020-11-19T16:06:03.798Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 1.234236726, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/uh0x0gFssG-000004", "remote": "10.0.10.20:43926"}

2020-11-19T16:06:13.626Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.144718788, "detections": 6, "device": null}

2020-11-19T16:06:13.627Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 1.163688418, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/uh0x0gFssG-000005", "remote": "10.0.10.20:59324"}

and this is the edgetpu:

2020-11-19T15:56:58.723Z    INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "default", "type": "tflite-edgetpu", "model": "models/jb_vb_edgetpu.tflite", "labels": 2, "width": 300, "height": 300}

2020-11-19T15:56:58.784Z    INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "default_edgetpu", "type": "tflite-edgetpu", "model": "models/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite", "labels": 80, "width": 300, "height": 300}

2020-11-19T15:57:00.237Z    INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "tensorflow", "type": "tensorflow", "model": "models/faster_rcnn_inception_v2_coco_2018_01_28.pb", "labels": 65, "width": -1, "height": -1}

2020-11-19T15:57:00.241Z    INFO    server/server.go:284    API Listening   {"package": "server", "address": ":8080", "tls": false, "version": "v0.2.5-0-gbf6d7a1-dirty"}

2020-11-19T15:57:02.895Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.02716341, "detections": 5, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}

2020-11-19T15:57:02.897Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 1.014697619, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/hmJsqN5QyH-000001", "remote": "10.0.10.20:56134"}

2020-11-19T15:57:12.697Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.008306565, "detections": 5, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}

2020-11-19T15:57:12.698Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 0.975427531, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/hmJsqN5QyH-000002", "remote": "10.0.10.20:43820"}

2020-11-19T15:57:22.755Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.008300731, "detections": 5, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}

2020-11-19T15:57:22.757Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 1.004568588, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/hmJsqN5QyH-000003", "remote": "10.0.10.20:58616"}

2020-11-19T15:57:32.754Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.00832712, "detections": 6, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}

2020-11-19T15:57:32.756Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 1.03227084, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/hmJsqN5QyH-000004", "remote": "10.0.10.20:45378"}

2020-11-19T15:57:42.750Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.008319898, "detections": 6, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}

2020-11-19T15:57:42.751Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 1.008720694, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/hmJsqN5QyH-000005", "remote": "10.0.10.20:34080"}

2020-11-19T15:57:52.709Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.009942682, "detections": 6, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}

2020-11-19T15:57:52.710Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 0.977224142, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/hmJsqN5QyH-000006", "remote": "10.0.10.20:46806"}

2020-11-19T15:58:02.865Z    INFO    tflite/detector.go:393  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.0097695, "detections": 5, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}

is there something that i am missing? the detection speed is almost identical in both cases and there is very little increase in detection speed, let alone 10 times increase i had with those 2 models. that does not make any sense to me cause i know edgetpu model should run much faster with google edgetpu usb thingy...

i did of course allowed detection to run for hours in both cases, so the speed did not increase/decrease noticeably and still stayed in the range of 1 second.

edit:: i think it's worth saying that i am running this on raspberry pi4, 2GB version with the official docker image.

[snowzach]

It looks to me like the bulk of your operation is not the detection but the actual HTTP request. The CPU is taking about 100ms and the EdgeTPU is taking about 10ms so the EdgeTPU is about 10 times faster.

What size image are you passing to doods? If it's not the same size as your detector (300x300) then doods resizes the image and then converts it to a raw format that Tensoflow can use.

1 second does seem kind of slow when the actual detection is quite fast. Is this maybe on a slow wifi connection as well?

[balonchiks]

thanks for you reply, i am passing 640x640 images so indeed looks like there is a conversion. the raspbery is connected via 5Ghz wifi and is placed less than a meter from the router so this should not be an issue, but i will give running on cable a try.

i am passing images from webcam feed from home assistant. any idea how can i easily resize them before sending to doods?

[balonchiks]

tried ethernet as opposed to wifi - no changes speedwise at all, same ~1 second.

is there anything else i can do apart from sending resized pics to decrease the http request time maybe?

[snowzach]

Yeah, I just fired up my Pi 4 as well and see the same.. I need to do some profiling and see where it's taking the time. I don't hear of a ton of people doing as rapid fire as that.

[balonchiks]

thanks, good to hear it's not mine config. leaving the issue opened and hoping you'll manage to find something

[snowzach]

Okay... decoding is super fast, resizing the image is taking about 700ms and converting it to the format that tensorflow understands is taking another 300ms. This is absurdly long. I'm going to look into other methods.

[snowzach]

@balonchiks try it now (pull the latest, I uploaded a fix last night)

[balonchiks]

awesome, thanks, all works and now and takes < 0.05 seconds:

2020-11-25T21:07:19.743Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 0.034647928, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/8P4SkovRna-000004", "remote": "10.0.10.20:46156"}
2020-11-25T21:07:29.495Z    INFO    tflite/detector.go:399  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.03528125, "detections": 3, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}
2020-11-25T21:07:29.497Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 0.04149853, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/8P4SkovRna-000005", "remote": "10.0.10.20:60426"}
2020-11-25T21:07:39.454Z    INFO    tflite/detector.go:399  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.031146646, "detections": 2, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}
2020-11-25T21:07:39.457Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 0.040029161, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/8P4SkovRna-000006", "remote": "10.0.10.20:46042"}
2020-11-25T21:07:49.492Z    INFO    tflite/detector.go:399  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.031020595, "detections": 2, "device": {"Type":1,"Path":"/sys/bus/usb/devices/2-2"}}
2020-11-25T21:07:49.494Z    INFO    server/server.go:139    HTTP Request    {"status": 200, "took": 0.036390915, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "aa09eeaee375/8P4SkovRna-000007", "remote": "10.0.10.20:58668"}