home-assistant / supervisor

:house_with_garden: Home Assistant Supervisor
https://home-assistant.io/hassio/
Apache License 2.0
1.71k stars 622 forks source link

Can't install addons: layer does not exist #5177

Closed myxor closed 4 weeks ago

myxor commented 1 month ago

Describe the issue you are experiencing

When trying to install "openwakeword" or "whisper" addon from the official addon store i am getting this error:

Can't install homeassistant/aarch64-addon-piper:1.5.0: 500 Server Error for http+docker://localhost/v1.45/images/homeassistant/aarch64-addon-piper:1.5.0/json: Internal Server Error ("layer does not exist")

As suggested at several places on the internet i ran ha supervisor repair which exited successful but the above problem remains.

From my supervisor log this part can see the whole process:

24-07-08 13:26:09.777 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/repair access from a0d7b954_ssh 2024-07-08 13:26:09.777 INFO (MainThread) [supervisor.core] Starting repair of Supervisor Environment 2024-07-08 13:26:09.779 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale containers 2024-07-08 13:26:09.789 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale images 2024-07-08 13:26:12.361 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale builds 2024-07-08 13:26:12.395 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale volumes 2024-07-08 13:26:12.405 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale networks 2024-07-08 13:26:12.416 INFO (SyncWorker_2) [supervisor.docker.manager] Fix stale container on hassio network 2024-07-08 13:26:12.494 INFO (SyncWorker_2) [supervisor.docker.manager] Fix stale container on host network 2024-07-08 13:26:12.839 INFO (MainThread) [supervisor.addons.manager] Found 2 add-ons to repair 2024-07-08 13:26:12.839 INFO (MainThread) [supervisor.addons.manager] Repairing for add-on: a0d7b954_sqlite-web 2024-07-08 13:26:12.841 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/hassio-addons/sqlite-web/aarch64 with tag 4.1.2. 2024-07-08 13:26:28.318 INFO (MainThread) [supervisor.addons.manager] Repairing for add-on: core_whisper 2024-07-08 13:26:28.320 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-whisper with tag 2.1.2. 2024-07-08 13:27:35.655 INFO (MainThread) [supervisor.supervisor] Repairing Supervisor 2024.06.2 2024-07-08 13:27:35.734 INFO (MainThread) [supervisor.core] Finished repair of Supervisor Environment 2024-07-08 13:27:47.571 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from a0d7b954_ssh 2024-07-08 13:28:01.370 INFO (MainThread) [supervisor.addons.addon] No core_piper addon Docker image homeassistant/aarch64-addon-piper found 2024-07-08 13:28:01.371 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-piper with tag 1.5.0. 2024-07-08 13:28:02.889 ERROR (MainThread) [supervisor.docker.interface] Can't install homeassistant/aarch64-addon-piper:1.5.0: 500 Server Error for http+docker://localhost/v1.45/images/homeassistant/aarch64-addon-piper:1.5.0/json: Internal Server Error ("layer does not exist") 2024-07-08 13:28:02.920 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-piper with tag 1.5.0. 2024-07-08 13:28:04.039 ERROR (MainThread) [supervisor.docker.interface] Can't install homeassistant/aarch64-addon-piper:1.5.0: 500 Server Error for http+docker://localhost/v1.45/images/homeassistant/aarch64-addon-piper:1.5.0/json: Internal Server Error ("layer does not exist") 2024-07-08 13:28:09.016 INFO (MainThread) [supervisor.addons.addon] No core_openwakeword addon Docker image homeassistant/aarch64-addon-openwakeword found 2024-07-08 13:28:09.017 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-openwakeword with tag 1.10.0. 2024-07-08 13:28:10.177 ERROR (MainThread) [supervisor.docker.interface] Can't install homeassistant/aarch64-addon-openwakeword:1.10.0: 500 Server Error for http+docker://localhost/v1.45/images/homeassistant/aarch64-addon-openwakeword:1.10.0/json: Internal Server Error ("layer does not exist") 2024-07-08 13:28:10.210 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-openwakeword with tag 1.10.0. 2024-07-08 13:28:11.321 ERROR (MainThread) [supervisor.docker.interface] Can't install homeassistant/aarch64-addon-openwakeword:1.10.0: 500 Server Error for http+docker://localhost/v1.45/images/homeassistant/aarch64-addon-openwakeword:1.10.0/json: Internal Server Error ("layer does not exist") 2024-07-08 13:28:14.576 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from a0d7b954_ssh

I also tried to pull the image via docker:

docker pull homeassistant/aarch64-addon-piper:1.5.0 1.5.0: Pulling from homeassistant/aarch64-addon-piper Digest: sha256:40c529021d55375057bdbe179de5abe701387dd53fabfcac9fa4ee9eefb23eca Status: Image is up to date for homeassistant/aarch64-addon-piper:1.5.0 docker.io/homeassistant/aarch64-addon-piper:1.5.0

and tried to delete the image via docker:

docker rmi homeassistant/aarch64-addon-piper:1.5.0 -f
Error response from daemon: unrecognized image ID sha256:50c62f02f1905940dd62ad997302464112d29c81832aa725b8e9e59c778184ce

but the above problem remains.

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

  1. Try to install piper or openwakeword addon
  2. Installation aborts with error "Internal Server Error ("layer does not exist")"

Anything in the Supervisor logs that might be useful for us?

24-07-08 13:26:09.777 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/repair access from a0d7b954_ssh
2024-07-08 13:26:09.777 INFO (MainThread) [supervisor.core] Starting repair of Supervisor Environment
2024-07-08 13:26:09.779 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale containers
2024-07-08 13:26:09.789 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale images
2024-07-08 13:26:12.361 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale builds
2024-07-08 13:26:12.395 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale volumes
2024-07-08 13:26:12.405 INFO (SyncWorker_2) [supervisor.docker.manager] Prune stale networks
2024-07-08 13:26:12.416 INFO (SyncWorker_2) [supervisor.docker.manager] Fix stale container on hassio network
2024-07-08 13:26:12.494 INFO (SyncWorker_2) [supervisor.docker.manager] Fix stale container on host network
2024-07-08 13:26:12.839 INFO (MainThread) [supervisor.addons.manager] Found 2 add-ons to repair
2024-07-08 13:26:12.839 INFO (MainThread) [supervisor.addons.manager] Repairing for add-on: a0d7b954_sqlite-web
2024-07-08 13:26:12.841 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/hassio-addons/sqlite-web/aarch64 with tag 4.1.2.
2024-07-08 13:26:28.318 INFO (MainThread) [supervisor.addons.manager] Repairing for add-on: core_whisper
2024-07-08 13:26:28.320 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-whisper with tag 2.1.2.
2024-07-08 13:27:35.655 INFO (MainThread) [supervisor.supervisor] Repairing Supervisor 2024.06.2
2024-07-08 13:27:35.734 INFO (MainThread) [supervisor.core] Finished repair of Supervisor Environment
2024-07-08 13:27:47.571 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from a0d7b954_ssh
2024-07-08 13:28:01.370 INFO (MainThread) [supervisor.addons.addon] No core_piper addon Docker image homeassistant/aarch64-addon-piper found
2024-07-08 13:28:01.371 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-piper with tag 1.5.0.
2024-07-08 13:28:02.889 ERROR (MainThread) [supervisor.docker.interface] Can't install homeassistant/aarch64-addon-piper:1.5.0: 500 Server Error for http+docker://localhost/v1.45/images/homeassistant/aarch64-addon-piper:1.5.0/json: Internal Server Error ("layer does not exist")
2024-07-08 13:28:02.920 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-piper with tag 1.5.0.
2024-07-08 13:28:04.039 ERROR (MainThread) [supervisor.docker.interface] Can't install homeassistant/aarch64-addon-piper:1.5.0: 500 Server Error for http+docker://localhost/v1.45/images/homeassistant/aarch64-addon-piper:1.5.0/json: Internal Server Error ("layer does not exist")
2024-07-08 13:28:09.016 INFO (MainThread) [supervisor.addons.addon] No core_openwakeword addon Docker image homeassistant/aarch64-addon-openwakeword found
2024-07-08 13:28:09.017 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-openwakeword with tag 1.10.0.
2024-07-08 13:28:10.177 ERROR (MainThread) [supervisor.docker.interface] Can't install homeassistant/aarch64-addon-openwakeword:1.10.0: 500 Server Error for http+docker://localhost/v1.45/images/homeassistant/aarch64-addon-openwakeword:1.10.0/json: Internal Server Error ("layer does not exist")
2024-07-08 13:28:10.210 INFO (MainThread) [supervisor.docker.interface] Downloading docker image homeassistant/aarch64-addon-openwakeword with tag 1.10.0.
2024-07-08 13:28:11.321 ERROR (MainThread) [supervisor.docker.interface] Can't install homeassistant/aarch64-addon-openwakeword:1.10.0: 500 Server Error for http+docker://localhost/v1.45/images/homeassistant/aarch64-addon-openwakeword:1.10.0/json: Internal Server Error ("layer does not exist")
2024-07-08 13:28:14.576 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from a0d7b954_ssh

System Health information

System Information

version core-2024.7.1
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.4
os_name Linux
os_version 6.6.31-haos-raspi
arch aarch64
timezone Europe/Berlin
config_dir /config
Home Assistant Community Store GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 5000 Installed Version | 1.34.0 Stage | running Available Repositories | 1387 Downloaded Repositories | 35
Home Assistant Cloud logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Home Assistant Supervisor host_os | Home Assistant OS 12.4 -- | -- update_channel | stable supervisor_version | supervisor-2024.06.2 agent_version | 1.6.0 docker_version | 26.1.4 disk_total | 229.2 GB disk_used | 114.2 GB healthy | true supported | true host_connectivity | true supervisor_connectivity | true ntp_synchronized | true virtualization | board | yellow supervisor_api | ok version_api | ok installed_addons | Advanced SSH & Web Terminal (18.0.0), NGINX Home Assistant SSL proxy (3.9.0), ESPHome (2024.6.6), File editor (5.8.0), Samba Backup (5.2.0), deCONZ (6.23.0), SQLite Web (4.1.2), Music Assistant (2.0.7), Whisper (2.1.2)
Dashboards dashboards | 4 -- | -- resources | 15 views | 36 mode | storage
Recorder oldest_recorder_run | 19. Mai 2024 um 02:14 -- | -- current_recorder_run | 8. Juli 2024 um 09:58 estimated_db_size | 10957.45 MiB database_engine | sqlite database_version | 3.45.3

Supervisor diagnostics

No response

Additional information

No response

agners commented 1 month ago

Did you try ha supervisor repair?

myxor commented 1 month ago

Did you try ha supervisor repair?

Yes i did as i wrote above. It can also be seen in the supervisor log that it repaired some issues but the above problem remains.

agners commented 1 month ago

Yes i did as i wrote above.

Whoops sorry about that, didn't read clearly :see_no_evil:

It can also be seen in the supervisor log that it repaired some issues but the above problem remains.

Ok, so yeah some of the Docker corruption unfortunately can't be fixed by Docker itself (and Supervisor essentially asks Docker to repair itself) :cry: . Your best way out here is to take a full backup, download it, and start over.

If you are adventurous, I've created a script once, it may fix your issue (see https://github.com/home-assistant/supervisor/issues/4738#issuecomment-1834627523). If you are using a data disk, I've also heard people being successful reviving their installation by detaching the disk, delete all content of the docker directory on the data partition, and boot the system again. The Supervisor will install the lastest version of Supervisor, Home Assistant Core etc. when doing this.

myxor commented 1 month ago

Thanks @agners, but your script wouldn't have worked as there is no 0 byte sized folder in /mnt/data/docker/image/overlay2/layerdb/sha256 on my installation.

Although i found a solution:

Try to remove the docker image:

docker rmi homeassistant/aarch64-addon-piper:1.5.0 -f
Error response from daemon: unrecognized image ID sha256:50c62f02f1905940dd62ad997302464112d29c81832aa725b8e9e59c778184ce

then with full root access go to folder /mnt/data/docker/image/overlay2/imagedb/content/sha256 and remove the file with the above hash (50c62f02f1905940dd62ad997302464112d29c81832aa725b8e9e59c778184ce).

try again to pull the image. it failed as follows:

docker pull homeassistant/aarch64-addon-piper:1.5.0  
1.5.0: Pulling from homeassistant/aarch64-addon-piper
25d3892798f8: Already exists 
4f4fb700ef54: Download complete 
e9518162dbce: Extracting [==================================================>]  10.42MB/10.42MB
15dd8bd13433: Download complete 
ae96d66eb1c6: Download complete 
failed to register layer: rename /mnt/data/docker/image/overlay2/layerdb/tmp/write-set-2938818620 /mnt/data/docker/image/overlay2/layerdb/sha256/0b34baae3cd2df852639e121a57562fdb234d298796d788dc44774ca9cafe8b3: file exists

I went to the folder /mnt/data/docker/image/overlay2/layerdb/sha256 and removed the folder with the hash from the previous output. I had to repeat these last to steps a couple of times but finally i was able to install piper from the addon store.

I hope this helps someone in the future :)

agners commented 4 weeks ago

I went to the folder /mnt/data/docker/image/overlay2/layerdb/sha256 and removed the folder with the hash from the previous output.

Do you remember what these locations were? Symlinks, folders or a file? I guess they corrupted somehow and Docker could not recover from that situation :cry:

It seems that this is somewhat known, this issue looks very similar: https://github.com/moby/moby/issues/44909.

I had to repeat these last to steps a couple of times but finally i was able to install piper from the addon store.

I hope this helps someone in the future :)

I guess it could be that there are still some other layers hanging. Let's hope not :crossed_fingers:

Thanks for the write-up, this might help someone else.

I am closing this here as this is a Docker issue/limitation and we can't fix this in Supervisor.