Closed joshdurbin closed 2 years ago
snowzach
commented
3 years ago That's a new one for me. Do you think it's possible the USB flash drive is bad? Maybe it starts swapping and causing the issue? I have a RPi4 with microsd and never had an issue. I thought I saw some comment about power... did you delete that? Could the power be sagging? I thought I saw the RPi4 needs a 3A power supply...
joshdurbin
commented
3 years ago Nah, I don't think it's an issue with the USB drive -- they're 2.5" western digital SSDs attached to UASP/USB3 to SATA3 interface adapters and I've got the exact same setup running on 4x RPI 4Bs -- all of which have been up and stable for months running various workloads. I think a relevant question is whether its something with the Firmware of the Coral device or the RPI4B. It's a bit interesting the device IDs change once loaded or probed via the app or whatever is happening there.
The PIs all have 3.5A power supplies.
I did write a thing about power earlier. I had installed the libedgetpu1-max package and replaced it with libedgetpu1-std thinking that it helped, but it didn't.
joshdurbin
commented
3 years ago I'm running these workloads as part of a Hashi Nomad, Consul, Vault cluster, with noobs having an affinity to land on a particular node that has the Coral Edge TPU. What I can do, maybe, is move that workload to the one RPI 3b+ I have that's got an SD card as it's boot/primary volume to see if it's able to do its thing. I'll try doing that later today.
joshdurbin
commented
3 years ago The PI3B+ doesn't have the same issues.
pi@rpi-3bplus-1:~ $ cat asdf.config
doods:
detectors:
- name: default
type: tflite
modelFile: models/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite
labelFile: models/coco_labels.txt
numThreads: 0
numConcurrent: 4
hwAccel: true
pi@rpi-3bplus-1:~ $ ls -lah models/
total 6.7M
drwxr-xr-x 2 pi docker 4.0K Nov 23 00:38 .
drwxr-xr-x 7 pi docker 4.0K Nov 23 09:41 ..
-rw-r--r-- 1 pi docker 930 Nov 23 00:38 coco_labels.txt
-rw-r--r-- 1 pi docker 6.7M Nov 23 00:38 ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tfliteIt looks like we're doing requests pretty quick with the TPU:
pi@rpi-3bplus-1:~ $ docker run -it --device /dev/bus/usb -v /home/pi/models/:/opt/doods/models -v /home/pi/asdf.config:/opt/doods/config.yaml -p 8080:8080 snowzach/doods:latest
2020-11-23T18:01:34.701Z INFO detector/detector.go:79 Configured Detector {"package": "detector", "name": "default", "type": "tflite-edgetpu", "model": "models/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite", "labels": 80, "width": 300, "height": 300}
2020-11-23T18:01:34.707Z INFO server/server.go:284 API Listening {"package": "server", "address": ":8080", "tls": false, "version": "v0.2.5-0-gbf6d7a1-dirty"}Logs like:
2020-11-23T18:02:41.632Z INFO detector/detector.go:163 Stream Request {"package": "detector"}
2020-11-23T18:02:41.695Z INFO tflite/detector.go:393 Detection Complete {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.060813247, "detections": 20, "device": {"Type":1,"Path":"/sys/bus/usb/devices/1-1.3"}}... and a load average close to 1.
The same request stream using the non-TPU model is significantly slower.
pi@rpi-3bplus-1:~ $ docker run -it -p 8080:8080 snowzach/doods:latest
2020-11-23T18:04:00.308Z INFO detector/detector.go:79 Configured Detector {"package": "detector", "name": "default", "type": "tflite", "model": "models/coco_ssd_mobilenet_v1_1.0_quant.tflite", "labels": 80, "width": 300, "height": 300}
2020-11-23T18:04:03.725Z INFO detector/detector.go:79 Configured Detector {"package": "detector", "name": "tensorflow", "type": "tensorflow", "model": "models/faster_rcnn_inception_v2_coco_2018_01_28.pb", "labels": 65, "width": -1, "height": -1}
2020-11-23T18:04:03.728Z INFO server/server.go:284 API Listening {"package": "server", "address": ":8080", "tls": false, "version": "v0.2.5-0-gbf6d7a1-dirty"}log entries:
2020-11-23T18:04:54.708Z INFO detector/detector.go:163 Stream Request {"package": "detector"}
2020-11-23T18:04:54.827Z INFO tflite/detector.go:393 Detection Complete {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.117880229, "detections": 10, "device": null}...with a CPU load average closer to 3 proving that the TPU is about 2x as fast. Which is better than I thought it'd be considering it's USB 2.
I'm using this example to send 1080p h264 streams to doods. There might be a penalty downsampling the video frames too. Perhaps I should switch to lower res.
1ubuntuuser
commented
3 years ago Man these look good.... shame you can't buy them in Australia!
Hope you get it going!
joshdurbin
commented
3 years ago Currently looking at a lead here thinking that it might related to power safe issues with the firmware.
snowzach
commented
3 years ago Hmmm.. That's interesting. Sorry, I don't have a lot of other suggestions. Most problems I have seen with EdgeTPU generally seem to relate to power or the USB. It either works perfect or has random crashes on a particular hardware platform. I have a raspberry Pi 4 and I don't have issues with it though. There is another issue with resizing causing it to be really slow, I am working on that one. It should be even faster soon.
mmatesic01
commented
3 years ago @snowzach Hi, I also run doods on RPi4+Coral USB (seperate HW from Home Assistant) without any mayor issues. Except occuring warnings/errors in image processing seen in HA logs. "Updating doods image_processing took longer than the scheduled update interval 0:00:03" "Update for image_processing.doods_terasa fails"
Is working but sometimes processing delays are longer. process_tme is in 0.2s-0.35s area, each 3 s.
How is the progress with issue with resizing going? Thanks.
snowzach
commented
3 years ago @mmatesic01 I fixed the issue with resizing making it slow, that shaved off a good 500ms or so. That is def fixed. Make sure for the models using the EdgeTPU you have numThreads set to 1 and numConcurrent also set to 1. It might be trying to send more than one detection at a time through the TPU and causing issues. I think I also upgraded the edgeTPU library as well. I'll have to check that. Been a while since I looked. Are you def on the latest version? If so, I can try checking for an EdgeTPU upgrade.
mmatesic01
commented
3 years ago @snowzach Mine were numThreads: 0 and numConcurrent: 4. I corrected that now. My version is: {"version":"v0.2.6-0-gb2a1c53-dirty"}
mmatesic01
commented
3 years ago Hi, In your opinion which of two parallel camera source formats should I use 1280x720@15FPS or 640x360@7FPS in better processing result and lower load on CORAL/RPi CPU? Or it does not matter? Scan interval is 2 s.
From: Zach notifications@github.com Sent: Saturday, December 26, 2020 22:01 To: snowzach/doods doods@noreply.github.com Cc: mmatesic01 mmatesic@live.com; Mention mention@noreply.github.com Subject: Re: [snowzach/doods] coral edge TPU issues on RPI4 (#46)
@mmatesic01https://github.com/mmatesic01 I fixed the issue with resizing making it slow, that shaved off a good 500ms or so. That is def fixed. Make sure for the models using the EdgeTPU you have numThreads set to 1 and numConcurrent also set to 1. It might be trying to send more than one detection at a time through the TPU and causing issues. I think I also upgraded the edgeTPU library as well. I'll have to check that. Been a while since I looked. Are you def on the latest version? If so, I can try checking for an EdgeTPU upgrade.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/snowzach/doods/issues/46#issuecomment-751395759, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIZLSD6FPSRH4MXJJTT5QPDSWZFLBANCNFSM4T7DDLZQ.
snowzach
commented
3 years ago @mmatesic01 it doesn't matter which one you use... It will be resized to 300x300 because that's what the model requires. That's the latest version as well. Curious if those changes to the config help?
mmatesic01
commented
3 years ago Changes you sugested did not make much of an effect on silencing the errors in HA. But in general it does not effect the expected functionality.
sebirdman
commented
3 years ago I'm seeing a very similar issue to @joshdurbin here. Running RPI4 off of a SSD with a Coral Edge TPU. The Edge TPU is plugged into a USB hub that is not powered.
HOWEVER, After the first run, so far my system seems stable. for whatever reason, the first time i start doods i get stuck, the error:
2021-01-21T22:33:57.270Z ERROR detector/detector.go:74 Could not initialize detector edgetpu: could not initialize edgetpu /sys/bus/usb/devices/2-1.2 {"package": "detector"}Then after that, all is well and I haven't seen any crashes. Will see if the stability continues. Maybe if doods tried to initialize a few times, It would work without hitting this the first run?
EDIT: as I've run out of spots on my current USB hub, i'm going to grab a larger one that is powered separately. perhaps that'll help resolve this for me.
EDIT 2: Even after using a powered USB hub, i still see the same issues.
mmatesic01
commented
3 years ago @sebirdman Do you use RPi 4 explicitly for Coral USB/Doods? If that is so, I advise you to connect the Coral directly to USB 3.0 port on RPi with USB cable that Coral comes with. In that way RPi 4 has no issues powering the Coral USB. But applying proper power supply for RPi 4 also must be met. I had lower performance issues when Coral USB was on USB 2.0 port or using USB 3.0 hub on USB 3.0 port. If Coral USB heats up a bit the means that it uses full USB bandwidth and that it is working with 100%-ish performance.
sebirdman
commented
3 years ago @mmatesic01 I've got this RPi 4 running a whole bunch of things, including Coral USB/Doods.
I used to run a NUT server for my UPS on it, but noticed that the UPS usb was frequently disconnecting from my RPi. After removing that device, so far - all of my issues have went away. Even the strange error on first start of the Doods server.
Thanks for the suggestion, I've got another RPi4 that is currently unused. If i continue to have issues i'll report them here and will try switching to a dedicated RPi4 for Coral USB/Doods.
Edit: Also, Do you have your dedicated RPi running wireless or wired internet? I'm curious how the network connection would effect the processing time from HA to Doods. I've got a few RPi's already setup around my home doing other things that i could try as well, but they're all wifi.
snowzach
commented
2 years ago Please try with new Python DOODS if this is still and issue: https://github.com/snowzach/doods2
mmatesic01
commented
2 years ago Hello,
Did the instalation and ran docker image with additiional » -- device /dev/bus/usb «. You have a typo in your instructions abou this parameter.
Then I changed config.yaml in my container and copied model and label file for Edge Coral.
doods: detectors:
{"detectors":[{"name":"default","type":"tensorflow2","model":"models/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite","labels":["person","bicycle","car","motorcycle","airplane","bus","train","truck","boat","traffic light","fire hydrant","n/a","stop sign","parking meter","bench","bird","cat","dog","horse","sheep","cow","elephant","bear","zebra","giraffe","n/a","backpack","umbrella","n/a","n/a","handbag","tie","suitcase","frisbee","skis","snowboard","sports ball","kite","baseball bat","baseball glove","skateboard","surfboard","tennis racket","bottle","n/a","wine glass","cup","fork","knife","spoon","bowl","banana","apple","sandwich","orange","broccoli","carrot","hot dog","pizza","donut","cake","chair","couch","potted plant","bed","n/a","dining table","n/a","n/a","toilet","n/a","tv","laptop","mouse","remote","keyboard","cell phone","microwave","oven","toaster","sink","refrigerator","n/a","book","clock","vase","scissors","teddy bear","hair drier","toothbrush"],"width":0,"height":0}]}
I did try to test my stream address in your GUI, http://192.168.xxx.xxx:8080/, which is acessible. @.*** But regarding stream (with my rtsp address) test all I got is empty blue web screen.
BR MArko Od: @.> Poslano: nedelja, 02. januar 2022 15:43 Za: @.> Kp: @.>; @.> Zadeva: Re: [snowzach/doods] coral edge TPU issues on RPI4 (#46)
Please try with new Python DOODS if this is still and issue: https://github.com/snowzach/doods2
— Reply to this email directly, view it on GitHubhttps://github.com/snowzach/doods/issues/46#issuecomment-1003726340, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIZLSD33NC4FAEW7UU67OWTUUBQAZANCNFSM4T7DDLZQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>
Howdy. I'm not sure this is explicitly a thing with doods, but thought you might have some insight into what might be going on with doods and Coral. I'm running Coral attached to a RPI4B that has USB boot enabled and uses an external USB disk as its system drive (no SD card at all). The OS is debian/pios lite running arm64.
I'm trying to transition from CPU-based detection to the Coral tf lite models. To do so I used the script to download all the models then, with the following config:
... am running:
docker run -it --device /dev/bus/usb -e logger.level='debug' -v /home/pi/models/:/opt/doods/models -v /home/pi/asdf.config:/opt/doods/config.yaml -p 8080:8080 snowzach/doods:latest. The first time through it errors out, stating it can't initialize the edgetpu.Prior to this run the
lsusboutput returns:...after the run, though, the
Bus 002 Device 003changes to:A second pass at running the container yields the following:
and will remain up and online for about another 8-10 seconds before the host OS crashes. It crashes because the USB host controller seems to die and hang. Specifically:
and:
Full log:
This requires a hard reset of the PI4. Any idea what could be going on here?
I've tried the Coral device on the 2 other RPI4Bs that I have and it produces the same error/problem.