av / harbor

Effortlessly run LLM backends, APIs, frontends, and services with one command.
https://github.com/av/harbor
Apache License 2.0
362 stars 17 forks source link

Auxiliary containers left running after harbor down #37

Open FrantaNautilus opened 4 days ago

FrantaNautilus commented 4 days ago

When command harbor down librechat or harbor down bionicgpt is issued the "auxiliary" containers, such as harbor.lc-db harbor.lc-vector harbor.lc-search for Librechat are left running according to docker ps. Is this expected behavior? I could not find details in the documentation. I also noticed that these containers are not being removed upon harbor down and are still listed in docker containers list --all, unlike the "main" containers of the respective frontends.


I am running harbor version 1.35, on Bluefin DX (Fedora Silverblue 40).

av commented 4 days ago

Thanks for trying things out!

No, it's not expected for those containers to remain running after harbor down, however that could happen if the down is cancelled prematurely.

I just tried following workflow locally and it doesn't seem to leave dangling containers:

 everlier@pop-os:~$ ▼ h ps
NAME      IMAGE     COMMAND   SERVICE   CREATED   STATUS    PORTS

 everlier@pop-os:~$ ▼ h u librechat
[+] Running 8/8
 ✔ Network harbor_harbor-network  Created                                                                                0.0s 
 ✔ Container harbor.webui         Healthy                                                                                6.6s 
 ✔ Container harbor.lc-db         Healthy                                                                                1.1s 
 ✔ Container harbor.lc-search     Healthy                                                                                1.1s 
 ✔ Container harbor.lc-vector     Healthy                                                                                1.1s 
 ✔ Container harbor.ollama        Healthy                                                                                1.1s 
 ✔ Container harbor.lc-rag        Healthy                                                                                1.0s 
 ✔ Container harbor.librechat     Healthy                                                                                1.0s 

 everlier@pop-os:~$ ▼ h ps
NAME               IMAGE                                      COMMAND                  SERVICE     CREATED          STATUS                   PORTS
harbor.lc-db       mongo                                      "docker-entrypoint.s…"   lc-db       10 seconds ago   Up 9 seconds             27017/tcp
harbor.lc-search   getmeili/meilisearch:v1.7.3                "tini -- /bin/sh -c …"   lc-search   10 seconds ago   Up 9 seconds             7700/tcp
harbor.lc-vector   ankane/pgvector:latest                     "docker-entrypoint.s…"   lc-vector   10 seconds ago   Up 9 seconds             5432/tcp
harbor.librechat   ghcr.io/danny-avila/librechat-dev:latest   "/app/start_librecha…"   librechat   10 seconds ago   Up 9 seconds             3080/tcp, 0.0.0.0:33891->33891/tcp, :::33891->33891/tcp
harbor.ollama      ollama/ollama:latest                       "/bin/ollama serve"      ollama      10 seconds ago   Up 9 seconds             0.0.0.0:33821->11434/tcp, :::33821->11434/tcp
harbor.webui       ghcr.io/open-webui/open-webui:main         "/app/start_webui.sh"    webui       10 seconds ago   Up 9 seconds (healthy)   0.0.0.0:33801->8080/tcp, :::33801->8080/tcp

 everlier@pop-os:~$ ▼ h d
[+] Running 8/8
 ✔ Container harbor.webui         Removed                                                                                                                                         0.2s 
 ✔ Container harbor.ollama        Removed                                                                                                                                         0.2s 
 ✔ Container harbor.librechat     Removed                                                                                                                                        10.2s 
 ✔ Container harbor.lc-db         Removed                                                                                                                                         0.1s 
 ✔ Container harbor.lc-rag        Removed                                                                                                                                         0.0s 
 ✔ Container harbor.lc-search     Removed                                                                                                                                         0.1s 
 ✔ Container harbor.lc-vector     Removed                                                                                                                                         0.1s 
 ✔ Network harbor_harbor-network  Removed                                                                                                                                         0.2s 

 everlier@pop-os:~$ ▼ h ps
NAME      IMAGE     COMMAND   SERVICE   CREATED   STATUS    PORTS

 everlier@pop-os:~$ ▼ 

Could it be possible that you're Ctrl+Cing out of harbor down? In that instance, not all dependencies might be resolved and containers indeed could be left handling.

If this could be the case - I understand why you'd want to do that, as, librechat containers are left hanging on up/down for 10s, which is Docker's default timeout for unresponsive processes. Such waiting indicates incorrect setup for the service entrypoint, which is another thing that could be fixed here.

FrantaNautilus commented 3 days ago

Thank you for helping me with this issue. Trying this again and I found that the harbor down works as expected, however its the command harbor down <handle> which leaves containers running. The containers remain running even after 56 minutes.

$harbor down librechat
[+] Running 2/1
 ✔ Container harbor.librechat     Removed                                                                                                                                                                                                                        10.3s 
 ! Network harbor_harbor-network  Resource is still in use
$ harbor ps
NAME               IMAGE                                COMMAND                  SERVICE     CREATED          STATUS                    PORTS
harbor.lc-db       mongo                                "docker-entrypoint.s…"   lc-db       56 minutes ago   Up 56 minutes             27017/tcp
harbor.lc-search   getmeili/meilisearch:v1.7.3          "tini -- /bin/sh -c …"   lc-search   56 minutes ago   Up 56 minutes             7700/tcp
harbor.lc-vector   ankane/pgvector:latest               "docker-entrypoint.s…"   lc-vector   56 minutes ago   Up 56 minutes             5432/tcp
harbor.ollama      ollama/ollama:latest                 "/bin/ollama serve"      ollama      56 minutes ago   Up 56 minutes             0.0.0.0:33821->11434/tcp, :::33821->11434/tcp
harbor.webui       ghcr.io/open-webui/open-webui:main   "/app/start_webui.sh"    webui       56 minutes ago   Up 56 minutes (healthy)   0.0.0.0:33801->8080/tcp, :::33801->8080/tcp

Only after running harbor down all containers were stopped and removed. Tested on CLI version: 0.2.0

av commented 3 days ago

Indeed, this is a bug in how the containers to stop are resolved, thank you for additional tests!

lee-b commented 2 days ago

Is this not due to docker compose up rather than docker compose up --remove-orphans (or the compose down equivalent)?