inveniosoftware / invenio-cli

CLI module for Invenio
https://invenio-cli.readthedocs.io
MIT License
10 stars 43 forks source link

invenio-cli services destroy does not clean pgadmin #338

Closed alejandromumo closed 1 year ago

alejandromumo commented 1 year ago

from the draft (@ppanero) :

As a consequence it does not clean the networks, and volumes

$ invenio-cli services destroy && invenio-cli services start

Destroying services' containers, volumes...
Destroying containers...
Failed to destroy services' containers.
Errors: Stopping zenodo-rdm_cache_1                 ...
Stopping zenodo-rdm_search_1                ...
Stopping zenodo-rdm_db_1                    ...
Stopping zenodo-rdm_opensearch-dashboards_1 ...
Stopping zenodo-rdm_mq_1                    ...
Stopping zenodo-rdm_opensearch-dashboards_1 ... done
Stopping zenodo-rdm_db_1                    ... done
Stopping zenodo-rdm_cache_1                 ... done
Stopping zenodo-rdm_search_1                ... done
Stopping zenodo-rdm_mq_1                    ... done
Found orphan containers (zenodo-rdm_pgadmin_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Removing zenodo-rdm_cache_1                 ...
Removing zenodo-rdm_search_1                ...
Removing zenodo-rdm_db_1                    ...
Removing zenodo-rdm_opensearch-dashboards_1 ...
Removing zenodo-rdm_mq_1                    ...
Removing zenodo-rdm_opensearch-dashboards_1 ... done
Removing zenodo-rdm_search_1                ... done
Removing zenodo-rdm_db_1                    ... done
Removing zenodo-rdm_mq_1                    ... done
Removing zenodo-rdm_cache_1                 ... done
Removing network zenodo-rdm_default
error while removing network: network zenodo-rdm_default id f1efd9eb732a487a9732dc3d10512f614933ced02b932244e92e43f1844940cb has active endpoints
$ docker ps -a

docker ps -a
CONTAINER ID   IMAGE                COMMAND            CREATED          STATUS          PORTS                                         NAMES
b9ceb2024c8c   dpage/pgadmin4:5.2   "/entrypoint.sh"   29 minutes ago   Up 29 minutes   0.0.0.0:5050->80/tcp, 0.0.0.0:5051->443/tcp   zenodo-rdm_pgadmin_1
alejandromumo commented 1 year ago

TL;DR: pgadmin container is missing from docker-compose.full.yml. Explanation below.


The issue starts here:

Found orphan containers (zenodo-rdm_pgadmin_1) for this project. 
If you removed or renamed this service in your compose file,
 you can run this command with the --remove-orphans flag to clean it up.

Issue seems to be related with running docker compose --file docker-compose.full.yml down --volumes

If ran using the default docker-compose.yml file ( docker compose --file docker-compose.yml down --volumes) yields no error.

I've noticed that we run docker compose --file docker-compose.yml up on start, however we run docker compose --file docker-compose.full.yml on destroy. This mismatch on compose files might be the origin of this issue.

I find it weird that just popped now, perhaps a configuration changed. services setup uses the default compose file because of the flag local=True on services commands (see source code) . However destroy commands are independent of such flag ( see source code)

We could use the flag --remove-orphans but I think it might just be hiding an underlying issue.

Running compose down with docker-compose.yml

Note that it removes zenodo-rdm_pgadmin_1 and removes 7 containers (6 containers and 1 network)

[+] Running 7/7
 ⠿ Container zenodo-rdm_pgadmin_1                Removed                                                                                                                        0.7s
 ⠿ Container zenodo-rdm_db_1                     Removed                                                                                                                        0.5s
 ⠿ Container zenodo-rdm_cache_1                  Removed                                                                                                                        0.6s
 ⠿ Container zenodo-rdm_mq_1                     Removed                                                                                                                        6.3s
 ⠿ Container zenodo-rdm_opensearch-dashboards_1  Removed                                                                                                                        0.6s
 ⠿ Container zenodo-rdm_search_1                 Removed                                                                                                                        0.8s
 ⠿ Network zenodo-rdm_default                    Removed                                                                                                                        0.1s

Running compose down with docker-compose.full.yml

Note it did not remove zenodo-rdm_pgadmin_1 and fails when removing the network.

[+] Running 5/5
 ⠿ Container zenodo-rdm_opensearch-dashboards_1  Removed                                                                                                                        0.4s
 ⠿ Container zenodo-rdm_mq_1                     Removed                                                                                                                        6.3s
 ⠿ Container zenodo-rdm_search_1                 Removed                                                                                                                        0.8s
 ⠿ Container zenodo-rdm_cache_1                  Removed                                                                                                                        0.3s
 ⠿ Container zenodo-rdm_db_1                     Removed                                                                                                                        0.3s
 ⠿ Network zenodo-rdm_default                    Error                                                                                                                          0.0s
failed to remove network 2d78e26151fd8ab7aa66db8bb0d4a578f57a82e847e10784badce1c5a75c7e44: Error response from daemon: error while removing network: network zenodo-rdm_default id 2d78e26151fd8ab7aa66db8bb0d4a578f57a82e847e10784badce1c5a75c7e44 has active endpoints

UPDATE

pgadmin was manually added to docker-compose.yml but not to docker-compose.full.yml. Therefore, pgadmin container is created on start but can't be destroyed since it's not declared in docker-compose.full.yml. Therefore, when removing network zenodo-rdm_default, pgadmin is thought to be orphan.

With the new configuration :

$ icli services destroy
Destroying services' containers, volumes...
Destroying containers...
Updating service setup status (False)...
Service setup status updated (new value False).
Services' containers destroyed.
alejandromumo commented 1 year ago

moved issue to zenodo-rdm: https://github.com/zenodo/zenodo-rdm/issues/166

ppanero commented 1 year ago

I find it weird that just popped now

I think I'm one of the few using the destroy command to cleanup 😅