search

abiosoft / colima

Container runtimes on macOS (and Linux) with minimal setup
MIT License
18.93k stars 382 forks source link

Trying to start a project in DDEV but getting an API error (500) #883

Open rpkoller opened 10 months ago

rpkoller commented 10 months ago

Description

I ran into something odd. I had a project running with ddev and colima 0.6.2 (on a m1pro with sonoma and vz/virtiofs). i wanted to start a second project in the terminal within vscode. i got the following output: 

$> ddev start
Starting modi... 
Failed to RunSimpleContainer to inspect database version/type: failed to create/start Docker container ({GetExistingDBType-modi-MxdvNY 0x14000220960 0x14000080480 <nil> <nil>}):API error (500): mkdir /var/lib/docker/overlay2/a784aef8fbf3af0f2a38904f816e31b88e268056b829a6b35028d77864d4a4c2-init: read-only file system, output=

when i checked the containers: 

$> docker ps -a
CONTAINER ID   IMAGE                                                 COMMAND                  CREATED             STATUS                       PORTS                                                                                                                                              NAMES
11c62cb4fdd6   ddev/ddev-traefik-router:v1.22.4                      "/entrypoint.sh --co…"   About an hour ago   Up About an hour (healthy)   127.0.0.1:80->80/tcp, 127.0.0.1:443->443/tcp, 127.0.0.1:8025-8026->8025-8026/tcp, 127.0.0.1:8036-8037->8036-8037/tcp, 127.0.0.1:10999->10999/tcp   ddev-router
afa5b0a2bd88   phpmyadmin:5                                          "/docker-entrypoint.…"   About an hour ago   Up About an hour             80/tcp                                                                                                                                             ddev-tests-phpmyadmin
6eeda14e67e0   ddev/ddev-dbserver-mariadb-10.5:v1.22.4-tests-built   "/docker-entrypoint.…"   About an hour ago   Up About an hour (healthy)   127.0.0.1:32777->3306/tcp                                                                                                                          ddev-tests-db
120c00fe7e45   ddev/ddev-webserver:20231108_new_lagoon-tests-built   "/pre-start.sh"          About an hour ago   Up About an hour (healthy)   8025/tcp, 127.0.0.1:32776->80/tcp, 127.0.0.1:32775->443/tcp                                                                                        ddev-tests-web
c71301f7966d   ddev/ddev-ssh-agent:v1.22.4-built                     "/entry.sh ssh-agent"    About an hour ago   Up About an hour (healthy)                                                                                                                                                      ddev-ssh-agent```

the container listed on the failed startup for the project modi doesnt exist. i then tried to poweroff ddev

$> ddev poweroff Failed removing custom certs/traefik configuration: failed to create/start Docker container ({poweroff-tHzHOH 0x1400021e1e0 0x1400020c480 }):API error (500): mkdir /var/lib/docker/overlay2/849ce4583dc2cf404267058b8b8952b57615930fd513a63748ba6eb4bdd1ace1-init: read-only file system Container ddev-tests-phpmyadmin Stopped Container ddev-tests-web Stopped Error response from daemon: container afa5b0a2bd88aac381e507e835604d2282c4fce285ee43807c4c295c4841fc79: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3ab71b22bcec4d2a8dcdd782d71ef9981c9931f87f8e22aaf499c48e8b57a2fb: read-only file system Failed to docker-compose down: composeCmd failed to run 'COMPOSE_PROJECT_NAME=ddev-tests docker-compose -f /Users/rkoller/Sites/tests/.ddev/.ddev-docker-compose-full.yaml down', action='[]', err='exit status 1', stdout='', stderr=' Container ddev-tests-phpmyadmin Stopping Container ddev-tests-web Stopping Container ddev-tests-phpmyadmin Stopped Container ddev-tests-phpmyadmin Removing Container ddev-tests-phpmyadmin Error while Removing Container ddev-tests-web Stopped Container ddev-tests-web Removing Container ddev-tests-web Error while Removing Error response from daemon: container afa5b0a2bd88aac381e507e835604d2282c4fce285ee43807c4c295c4841fc79: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3ab71b22bcec4d2a8dcdd782d71ef9981c9931f87f8e22aaf499c48e8b57a2fb: read-only file system' Removing container: ddev-tests-phpmyadmin Failed to stop project tests: could not remove container ddev-tests-phpmyadmin: API error (500): container afa5b0a2bd88aac381e507e835604d2282c4fce285ee43807c4c295c4841fc79: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3ab71b22bcec4d2a8dcdd782d71ef9981c9931f87f8e22aaf499c48e8b57a2fb: read-only file system


i then tried a hard reset:

$> docker rmi -f $(docker images -q) Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3892998772: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json1151577697: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3079715760: read-only file system Error response from daemon: conflict: unable to delete 1066e79ef629 (cannot be forced) - image is being used by running container 6eeda14e67e0 Error response from daemon: conflict: unable to delete a8483ea26b94 (cannot be forced) - image is being used by running container c71301f7966d Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json1453494246: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3496979304: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3644803181: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3927001417: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json2578752554: read-only file system Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json3753705229: read-only file system Error response from daemon: conflict: unable to delete 97c388ff3350 (cannot be forced) - image is being used by running container 11c62cb4fdd6 Error response from daemon: open /var/lib/docker/image/overlay2/.tmp-repositories.json389738583: read-only file system

and

$> docker rm -f $(docker ps -aq) Error response from daemon: Could not kill running container 11c62cb4fdd60e4793b2ce078dec0633f1134a1487f06e6009018caff80c1952, cannot remove - container 11c62cb4fdd6 PID 284811 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes Error response from daemon: container afa5b0a2bd88aac381e507e835604d2282c4fce285ee43807c4c295c4841fc79: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3ab71b22bcec4d2a8dcdd782d71ef9981c9931f87f8e22aaf499c48e8b57a2fb: read-only file system Error response from daemon: Could not kill running container 6eeda14e67e032c9f59bba1c64a23cb80d17a4a082d65898113449619b8527cb, cannot remove - container 6eeda14e67e0 PID 280580 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes Error response from daemon: container 120c00fe7e45844a8c7860ce9d76ed41e891d4b7344665dbe8d466a99f472802: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/391d9432305b168f4a622f4927d74b55f046191a65201f2d27bbba67516fd78d: read-only file system Error response from daemon: container c71301f7966dfeaacda4248f1ce4dd2a98b0fe61d56fdae435ea531202bd0fca: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/d8fd6e612336f8b08c40b77cfa3afa9fbb49c3b6f1a5cd2cf756e64032d5750f: read-only file system```

i also tried to execute docker rm -f $(docker ps -aq)inside the colima vm after colima ssh same result like on the host.

no idea what caused it and or if you need any further informations?

Version

$> colima version && limactl --version && qemu-img --version
colima version 0.6.2
git commit: 22d7e5fbc86d5b8e3b27065a762800bc7960a0ff

runtime: docker
arch: aarch64
client: v24.0.7
server: v24.0.5
limactl version 0.18.0
qemu-img version 8.1.2
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
ddev version v1.22.4-37-gca6b43689

Operating System

Output of colima status

$> colima status
INFO[0000] colima is running using macOS Virtualization.Framework 
INFO[0000] arch: aarch64                                
INFO[0000] runtime: docker                              
INFO[0000] mountType: virtiofs                          
INFO[0000] socket: unix:///Users/rkoller/.colima/default/docker.sock

Reproduction Steps

I've outlined the steps that snowballed into the trouble. not sure if it is reproducible.

Expected behaviour

the error cascade happened out of the blue.

Additional context

No response

abiosoft commented 10 months ago

Can you try with the current development version?

brew install --HEAD colima
rfay commented 10 months ago

My bet is this is solved in 0.6.4

rpkoller commented 10 months ago

My bet is this is solved in 0.6.4

i think so too.

Can you try with the current development version?

testing might be difficult with the profile i ran into the issue. in the course of trying to recover or at least powerdown ddev i deleted some images and projects. and in the end the situation was worth. i've backup'ed and cloned the profile still based on the recommendation of @rfay but then deleted the original and created a new profile to being able to access some of the projects today. and i would have tested the HEAD version but now already a release is tagged and i gonna wait a few more hours until it shows up on homebrew and then give it a try. but as i said i also think the odds are high that things will be fixed with this release and i would also have a hard time to reproduce the issue anyway. i'Ve just reported it as a fyi that something like that happend and "maybe" you would have been able to notice any clues what has lead to that error. but will test 0.6.4 as soon as it is available on homebrew. thank you for the fast release!