robmarkcole / coral-pi-rest-server

Perform inferencing of tensorflow-lite models on an RPi with acceleration from Coral USB stick
https://coral.ai/products/accelerator
MIT License
67 stars 20 forks source link

coral flask server hangs after a while #40

Closed lmamakos closed 4 years ago

lmamakos commented 4 years ago

I've noticed that in my environment, the coral TPU stick goes into an unusual state and the flask server ceases responding to requests. I'm running this in a Docker container on Ubuntu 18.04 on Intel Core i5.

Of note that when it gets into this state, the white LED on the Coral TPU USB stick has this consistent blinking going on. I've tried to figure out what the LED state is supposed to represent, but haven't obviously found documentation to that effect. Any insight here would be useful.

I've added a healthcheck to the docker container, which successfully noticed that the server isn't responding to queries. Too bad it doesn't automatically restart the container :-) I think that I need to be running docker swarm or something a bit more sophisticated for the automatic restart...

Of course I'm away on travel, so debugging is limited. I need to sync-up versions of what I'm running with the most recent version here, and also look at some suspicious/weirdness in terms of what packages are being installed into the container by the coral install script. I fear there might be a mix of Python 3.6 and 3.7 stuff going on in there, and I'm not 100% clear on what's being used. Of course, I'd think that obvious dependency issues would be evident right away, not days later after the container has been running.

I wanted to just open an issue here as a place-keeper and also see if anyone else might have noticed similar behavior in an attempt to shorten up some debugging steps.

robmarkcole commented 4 years ago

Hi @lmamakos thanks for reporting, this is a duplicate of https://github.com/robmarkcole/coral-pi-rest-server/issues/38 I will add some more detail to that issue