search

codeproject / CodeProject.AI-ObjectDetectionCoral

4 stars 2 forks source link

ObjectDetectionCoral stops working when restarting the docker container. #7

Open mikegleasonjr opened 7 months ago

mikegleasonjr commented 7 months ago

Area of Concern

Describe the bug

  1. I am running a docker container with the codeproject/ai-server image with 2 volumes (so that I can restart it without having to reinstall/reconfigure everything):
    • /etc/codeproject/ai
    • /app/modules
  2. I installed the Object Detection (Coral) module, which worked fine with my TPU. I called the API and verified that everything was working.
  3. I stopped the container
  4. I started back the container
  5. The module now has this error: objectdetection_coral_adapter.py: An exception occurred initialising the module: libedgetpu.so.1: cannot open shared object file: No such file or dir

Expected behavior The module restarts properly in the docker container.

I know what the problem actually is: In ObjectDetectionCoral/install.sh, there is a line to copy the shared library in /usr/lib/x86_64-linux-gnu/ upon module installation:

cp "${moduleDirPath}/edgetpu_runtime/${edgetpu_folder}/k8/libedgetpu.so.1.0" "/usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0"

Obvisouly when restarting the container, this change is lost since I do not mount /usr/lib/x86_64-linux-gnu/ as a volume to be persistent across "reboots".

I don't know if modules have a startup hook where these steps of copying the shared object and running of ldconfig could be done there?

Screenshots N/A

Your System (please complete the following information):

Additional context See Expected behavior

matthewDDennis commented 7 months ago

How are you starting the Docker container?

mikegleasonjr commented 7 months ago

Hi @matthewDDennis, thanks for the help. It is my feeling that the bug report wasn't fully understood though. The bug is actually known and described in the "Expected behavior" section (at least I tried to explain it). This isn't about Docker not restarting the container.

Please feel free to ask questions if it is still misunderstood 🙏🏻

mikegleasonjr commented 7 months ago

Maybe the ObjectDetectionCoral module should redo some steps it does upon installation in the initialise phase of the module: https://www.codeproject.com/ai/docs/devguide/module_examples/add_python_module.html#writing-the-module

It copies files into the host at the location: /usr/lib/x86_64-linux-gnu/. Those files are obviously lost when restarting the container.

There is no mention about mounting other volumes other than the settings/module folders in the documentation: https://www.codeproject.com/ai/docs/install/running_in_docker.html#advanced-docker-launch-settings-saved-outside-of-the-container

galperinm commented 2 months ago

@mikegleasonjr How did you end up working around this? Did you have to mount usr/lib/x86_64-linux-gnu/ as a persistent volume?

mikegleasonjr commented 2 months ago

Mounting usr/lib/x86_64-linux-gnu/ as a persistent volume would mean going through hoops to first get the files from the image before the module installation since the directory would be first empty on the host (and then empty in the running container).

I felt the comprehensive bug report wasn't even read according to the comments I had so I felt helpless trying to make ppl understand what was happening and a waste of efforts to go to such an extent to file a bug report. I stopped using the module (and codeprojectai).

So yes I gave an example where some installation files were lost but I don't know if it involves a game of wack-a-mole and if something else would still be missing after a container restart. I think so, I think some apt install are being made upon installation that would be lost upon container restart.

Also the official doc says to mount the Docker image in a certain way so this would contradict what has to be done here to restore the context.

galperinm commented 2 months ago

Thanks for the update @mikegleasonjr. It's a shame this isn't under active development anymore given that such a major bug is unacknowledged half a year later, guess I'll need to move off of codeproject.ai as well. Surprising, I can't imagine you and I are the only two Coral + codeprojectai docker users out there.

If you feel like sharing, I'd love to know what you moved to. My intention was to use this with Frigate and Doubletake for facial recognition supporting Coral. Looks like the alternatives are dead too-- Deepstack is no longer being developed either (docker image last updated over 2 years ago), and Compreface likewise has almost a year since last update.

matthewDDennis commented 5 days ago

Installing the latest version of the Server, 2.9.0, should resolve the issue. Since 2.8.0 the Server will check to see if the container Id has changed and rerun the install scripts to ensure all the required files and libs are initialized.

In the case of the Docker image, you need to pull the appropriate codeproject/ai-server image that matches your CPU/GPU configuration. The latest image is 2.8.0, but a 2.9.0 should be available soon.