HA Supervisor Failed setup: troubles with Docker 25 upgrade. Supervisor unable to start

GitWally commented 9 months ago

Describe the issue you are experiencing

Today MY HA went crazy, probably after a Debian OS update: it all started after an OS reboot.

From Settings > Integrations I see "Home Assistant Supervisor Failed setup, will retry"

immagine

Several other integrations same behaviour, some other are working (Shelly, MQTT, ZHA, HomeKit ...)

immagine

From HA Core Logs I see (among others):

Logger: homeassistant.helpers.system_info
Source: helpers/system_info.py:72
First occurred: 18:12:57 (1 occurrences)
Last logged: 18:12:57

No Home Assistant Supervisor info available

Logger: homeassistant.components.hassio
Source: components/hassio/__init__.py:557
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 18:02:11 (4 occurrences)
Last logged: 18:17:21

Can't read Supervisor data:

If I try to read HA Supervisor Logs: Failed to get supervisor logs, 502: Bad Gateway

What could be happend?

In Debian I found this updates to Docker Packages, today (with command cat /var/log/apt/history.log):

Start-Date: 2024-01-19  17:14:36
Commandline: packagekit role='update-packages'
Upgrade: docker-compose-plugin:amd64 (2.21.0-1~debian.11~bullseye, 2.24.1-1~debian.11~bullseye), docker-ce-cli:amd64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye), docker-buildx-plugin:amd64 (0.11.2-1~debian.11~bullseye, 0.12.1-1~debian.11~bullseye), docker-ce:amd64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye), docker-ce-rootless-extras:amd64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye)
End-Date: 2024-01-19  17:14:49

It all began after a reboot I did one hour after Debian OS Updates.

Tried to reboot Debian and HA several times: no changes Tried to go for a backup: not visible when I press the button

Thanks

What type of installation are you running?

Home Assistant Supervised

Which operating system are you running on?

Debian

Steps to reproduce the issue

1. 2. 3. ...

Anything in the Supervisor logs that might be useful for us?

not available

System Health information

not available

Supervisor diagnostics

cannot find download diagnostics

Additional information

No response

cadsdf commented 9 months ago

commands to pass (under putty ??) for to downgrade docker CE

@titou4307 for me the following was sufficient to get everything working:

List available versions of docker-ce

apt list -a docker-ce

Select the full version name of the previous working one <version_name> and downgrade the package

sudo apt-get install -y --allow-downgrades docker-ce=<version_name>

so I used this command

sudo apt-get install -y --allow-downgrades docker-ce=5:24.0.7-1~debian.12~bookworm

Reboot the system

sudo reboot

Trebor87 commented 9 months ago

I have the same error in my HA on But I am frenchee and use google to translate....

Bonjour il faut juste faire un 'downgrade' de Docker

Indiqué dans les premiers posts

C'est très simple

titou4307 commented 9 months ago

@cadsdf and @Trebor87

Thanks for your help ... Merci pour le message en français aussi !!

If I understood correctly I also deactivated the docker update...

To avoid upgrading docker before the fixed version is released:

sudo apt-mark hold docker-compose-plugin docker-ce-cli docker-buildx-plugin docker-ce docker-ce-rootless-extras

After docker 25.01 has been released to upgrade docker again:

sudo -i
apt-mark unhold docker-compose-plugin docker-ce-cli docker-buildx-plugin docker-ce docker-ce-rootless-extras
apt update && sudo apt upgrade -y && sudo apt autoremove –y

Many thanks one more time And have a good year 2024

Trebor87 commented 9 months ago

@cadsdf and @Trebor87

Thanks for your help ... Merci pour le message en français aussi !!

If I understood correctly I also deactivated the docker update...

To avoid upgrading docker before the fixed version is released:

sudo apt-mark hold docker-compose-plugin docker-ce-cli docker-buildx-plugin docker-ce docker-ce-rootless-extras

After docker 25.01 has been released to upgrade docker again:
sudo -i
apt-mark unhold docker-compose-plugin docker-ce-cli docker-buildx-plugin docker-ce docker-ce-rootless-extras
apt update && sudo apt upgrade -y && sudo apt autoremove –y 
Many thanks one more time And have a good year 2024

Tu as tout compris, Soit bloquer les mises à jour, soit ne fais pas les mises a jours tant que ce n'est corrigé en version 25.01

Bonne année aussi, Pour info j'habite en France 👍

fireashes commented 9 months ago

It worked for me in Debian 12 bookworm

Thanks to all for your contributions: it worked also for me! let me summarize for those newbies (like me) who could find out to be in the same shoes.

1. Check apt log to see the most recet updates to docker With command cat /var/log/apt/history.log you can read which old package version was upgraded: you must go back to that one. Eg:

docker-compose-plugin:amd64 (**2.21.0-1~debian.11~bullseye**, 2.24.1-1~debian.11~bullseye),

means docker-compose-plugin package must be roled back to version 2.21.0-1~debian.11~bullseye with command:

sudo apt install docker-compose-plugin=2.21.0-1~debian.11~bullsey

2. Rollback to the previous version of Docker packages This was my apt log:
Commandline: packagekit role='update-packages'
Upgrade: 
docker-compose-plugin:amd64 (2.21.0-1~debian.11~bullseye, 2.24.1-1~debian.11~bullseye), 
docker-ce-cli:amd64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye), 
docker-buildx-plugin:amd64 (0.11.2-1~debian.11~bullseye, 0.12.1-1~debian.11~bullseye), 
docker-ce:amd64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye), 
docker-ce-rootless-extras:amd64 (5:24.0.7-1~debian.11~bullseye, 5:25.0.0-1~debian.11~bullseye)
this was the rollback of the same packages:
sudo apt install \
docker-compose-plugin=2.21.0-1~debian.11~bullseye \
docker-ce-cli=5:24.0.7-1~debian.11~bullseye \
docker-buildx-plugin=0.11.2-1~debian.11~bullseye \
docker-ce=5:24.0.7-1~debian.11~bullseye \
docker-ce-rootless-extras=5:24.0.7-1~debian.11~bullseye
3. Prevent the same packages to upgrade in the short term To avoid the same issue to happen, until a stable solution will be found I decided to prevent the same packages to upgrade, as suggested by fcastilloec
sudo apt-mark hold \
docker-compose-plugin \
docker-ce-cli \
docker-buildx-plugin \
docker-ce \
docker-ce-rootless-extras \
Anyhow I think this is a bug that should be solved: preventing Docker packages to upgrade is just a temporary workaround, in my humble opinion. Many thanks to all once again!

For debian baremetal the following was the result Finding Upgrades related to docker from history cat /var/log/apt/history.log The relevant result

docker-compose-plugin:amd64 (2.21.0-1~debian.12~bookworm, 2.24.1-1~debian.12~bookworm),
docker-ce-cli:amd64 (5:24.0.7-1~debian.12~bookworm, 5:25.0.0-1~debian.12~bookworm),
docker-buildx-plugin:amd64 (0.11.2-1~debian.12~bookworm, 0.12.1-1~debian.12~bookworm),
docker-ce:amd64 (5:24.0.7-1~debian.12~bookworm, 5:25.0.0-1~debian.12~bookworm),
docker-ce-rootless-extras:amd64 (5:24.0.7-1~debian.12~bookworm, 5:25.0.0-1~debian.12~bookworm)

Rolling back

sudo apt install \
docker-compose-plugin=2.21.0-1~debian.12~bookworm \
docker-ce-cli=5:24.0.7-1~debian.12~bookworm \
docker-buildx-plugin=0.11.2-1~debian.12~bookworm \
docker-ce=5:24.0.7-1~debian.12~bookworm \
docker-ce-rootless-extras=5:24.0.7-1~debian.12~bookworm

To hold the version of docker until there is a fix

sudo apt-mark hold \
docker-compose-plugin \
docker-ce-cli \
docker-buildx-plugin \
docker-ce \
docker-ce-rootless-extras

To unhold the version of docker after the fix in future

sudo apt-mark unhold \
docker-compose-plugin \
docker-ce-cli \
docker-buildx-plugin \
docker-ce \
docker-ce-rootless-extras

d4v1dk0n7 commented 9 months ago

Hello guys,

I spent a lot of hours today trying to figure out the cause of the issue, and thanks to you now it's solved!! In my case, I'm running HA in a raspberry 4, these are the steps I've followed to solve the issue:

1.- Check the previous versions: apt list -a -q docker-ce

2.- Downgrade the version installed: apt-get install -y --allow-downgrades docker-ce=5:24.0.7-1~debian.11~bullseye

3.- Reboot.

Now all is working great, but as a curiosity, when I check the docker version installed it's still showing the previous:

root@nas:~# docker --version Docker version 25.0.0, build e758fe5

vgisbert commented 9 months ago

Hello guys,

I spent a lot of hours today trying to figure out the cause of the issue, and thanks to you now it's solved!! In my case, I'm running HA in a raspberry 4, these are the steps I've followed to solve the issue:

1.- Check the previous versions: apt list -a -q docker-ce

2.- Downgrade the version installed: apt-get install -y --allow-downgrades docker-ce=5:24.0.7-1~debian.11~bullseye

3.- Reboot.

Now all is working great, but as a curiosity, when I check the docker version installed it's still showing the previous:

root@nas:~# docker --version Docker version 25.0.0, build e758fe5

Correct. But with the command: docker version It is seen that the Client is 25.0.0 but the Server is 24.0.7 So with that command it works perfectly. But updates should also be hold until they release a solution.

d4v1dk0n7 commented 9 months ago

Hello guys,

I spent a lot of hours today trying to figure out the cause of the issue, and thanks to you now it's solved!! In my case, I'm running HA in a raspberry 4, these are the steps I've followed to solve the issue:

1.- Check the previous versions: apt list -a -q docker-ce

2.- Downgrade the version installed: apt-get install -y --allow-downgrades docker-ce=5:24.0.7-1~debian.11~bullseye

3.- Reboot.

Now all is working great, but as a curiosity, when I check the docker version installed it's still showing the previous:

root@nas:~# docker --version Docker version 25.0.0, build e758fe5

Correct. But with the command: docker version It is seen that the Client is 25.0.0 but the Server is 24.0.7 So with that command it works perfectly. But updates should also be hold until they release a solution.

I didn't know that, thanks for the info!

ElektronikCZ commented 9 months ago

I had the same issue. I have webmin with autoupdate, and this happened sometime during the weekend. I figured supervisor updated, so i updated from debian 11 to debian 12 and updated OS agent to the latest version. Didnt work. Then i came across this issue and downgraded docker. Works perfectly now

titou4307 commented 9 months ago

@ElektronikCZ I use me too Webmin with autoupdate

I have hold the update of docker until a new docker updating who take the modifications

Adam1697 commented 9 months ago

Thank you all for the suggestions, downgrading docker solved the problem. Sorry for my long post, that was my very first in github ever.

Edocsyl commented 9 months ago

Thanks for the workaround, worked for my environment.

gulliver103 commented 9 months ago

Downgrading docker solved the problem for me too. Thanks

SergeiGer commented 9 months ago

sudo apt install \ docker-ce=5:24.0.7-1~ubuntu.22.04~jammy \ docker-ce-cli=5:24.0.7-1~ubuntu.22.04~jammy \ docker-buildx-plugin=0.11.2-1~ubuntu.22.04~jammy \ docker-compose-plugin=2.21.0-1~ubuntu.22.04~jammy \ docker-ce-rootless-extras=5:24.0.7-1~ubuntu.22.04~jammy

th3wingman commented 9 months ago

Same issue on ubuntu 20.04, can't assign static IP from outside the dynamic range. Downgrade to this:

ii  docker-ce                 5:24.0.7-1~ubuntu.20.04~focal amd64
ii  docker-ce-cli             5:24.0.7-1~ubuntu.20.04~focal amd64
ii  docker-ce-rootless-extras 5:24.0.7-1~ubuntu.20.04~focal amd64

works around the problem

lazarvgd commented 9 months ago

commands to pass (under putty ??) for to downgrade docker CE

@titou4307 for me the following was sufficient to get everything working:
* List available versions of `docker-ce`
apt list -a docker-ce
* Select the full version name of the previous working one `<version_name>` and downgrade the package
sudo apt-get install -y --allow-downgrades docker-ce=<version_name>
so I used this command
sudo apt-get install -y --allow-downgrades docker-ce=5:24.0.7-1~debian.12~bookworm
* Reboot the system
sudo reboot

thanks a lot, you saved the day!

sentry-io[bot] commented 9 months ago

Sentry issue: SUPERVISOR-68

donaldguy commented 9 months ago

Change to validation (moby/moby@3092b261e2d2379954194f7889170e808b434034) is ostensibly fixed upstream in moby/moby#47135

but with no clear immediate plan for them to cut a 25.0.1 release ; there is no direct workaround outside of downgrade

https://github.com/moby/moby/issues/47120#issuecomment-1904817005 :

@ngandrass No. Downgrading to v24 or waiting for v25.0.1 are your only options.

I suspect one might get away with manually creating a hassio network that widened the IPRange to the full "172.30.32.0/23" - but I'm not sure what knock-on effects that would have in terms of how e.g. add-ons are exposed?

One might also be able to make a e.g. hassio_core network (also) with subnet of 172.30.32.0/23 (or if docker reinforces non-overlap limit it to 172.30.32.0/24 and the hassio to 172.30.33.0/24) and/but-definitely range of 172.30.32.0/24

In my immediate testing I saw no signs of docker actually isolating the (hassio and default bridge (172.17.0.0/24) subnets, nor host and either), perhaps cause neither is marked internal. [It makes one wonder what the point really is.]

but if they were more locked down on other systems, there is probs some way to explicitly pair them

Datortarp5 commented 9 months ago

Debian 12, docker downgrade from 5:25.0.0-1 to 5:24.0.7-1 did not solve the issue at all.

Installed fresh Deb 12, installed Docker 5:24.0.7-1 - Supervisor wont set up anyway. Just only after ha jobs options --ignore-conditions healthy HA stated to work but anyway with critical error Unhealthy system - Docker misconfigure and Unsupported system - Unsupported software. Wanted to migrate from HassOS to Supervised by easy way, got endless surprises 🥲

donaldguy commented 9 months ago

@Datortarp5 I did:

systemctl stop hassio-supervisor
docker rm -f $(docker ps -aq)
docker network prune
systemctl stop docker
downgrade of docker-ce and docker-ce-cli
systemctl start hassio-supervisor (note that the supervised-installer package added a containerd volume mount to /usr/sbin/hassio-supervisor recently; not sure if thats needed)
ha supervisor repair
ha supervisor restart

more Broadly you should grab the stuff from home-assitant/supervised-installer and build and dpkg -i that deb

Tamsy commented 9 months ago

Wanted to migrate from HassOS to Supervised by easy way, got endless surprises

I advise you follow the guides for the RPI HERE or for any other device (Mini PC, NUC etc.) HERE.

Make sure you follow the guide step by step and by the book!

Take care to downgrade Docker as already posted here again and again after Section 1 and before Section 2 of the guide and you should be good.

emavarela commented 9 months ago

Debian Bookworm here as well.... you guys rock. Saved my install. Thanks!

SergeyFilippov commented 9 months ago

Debian Bookworm here as well.... you guys rock. Saved my install. Thanks!

Same story here. Almost started wiping the system and doing fresh install when found this thread. Downgrade helped well.

One question: why is this bug marked as "unsupported"?

Is supervised installation not one of the official ways to host HA? According to official docs my system is perfectly supported, and I just did my regular system update (what is also my "responsibility" according to the docs).

Datortarp5 commented 9 months ago

3. network prune

Looks like this is what has been missing :) Many thanks!!

dimkin-eu commented 9 months ago

this is out ( with potential fix ) https://github.com/moby/moby/releases/tag/v25.0.1

Is anyone brave enough to test? :)

churchofnoise commented 9 months ago

Testing, seems to work!

Trebor87 commented 9 months ago

this is out ( with potential fix ) https://github.com/moby/moby/releases/tag/v25.0.1

Is anyone brave enough to test? :)

Thanks for the heads up

Just ran update and upgrade

It works just fine 👍

dimkin-eu commented 9 months ago

tested also myself - it works without and with images killing/deleting, so it seems like we are safe and the ticket could be closed

MattWestb commented 9 months ago

Armbian Ubuntu have releasing the 25.01 for some hours ago (was not in the morning) and looks working OK with Supp on the first small RCP test system so not going for the large ZHA production and RCP test system.

Tamsy commented 9 months ago

For the impatient on Debian to find out whether 25.0.1 is already available from the Debian Repositories just run:

sudo apt-cache policy docker-ce docker-ce-cli docker-ce-rootless-extras

dMopp commented 9 months ago

Still some problems with some addons even using the new version :( (ebusd addon cant connect anymore to my ebusd adapter)

fragkopoulos commented 9 months ago

Upgraded to 25.0.1 on raspberry pi (25.0.1-1~debian.11~bullseye arm64). Everything is working!

davecoutts commented 9 months ago

I'm happy to report that my Home Assistant is operating without issue after upgrading to docker-ce 25.0.1 on debian 12 bookworm. I had to reboot the machine after unholding the docker packages and running apt upgrade in order to gain access to the Home Assistant. As per the commands below.

sudo apt-mark unhold \
docker-ce \
docker-ce-cli \
docker-buildx-plugin \
docker-compose-plugin \
docker-ce-rootless-extras

sudo apt update
sudo apt upgrade

sudo reboot

d4v1dk0n7 commented 9 months ago

Updated to 25.0.1 on raspberry 4, all working fine.

MattWestb commented 9 months ago

looks like 25.X is braking InfluxDB then all my systems is getting this:

ERROR (influxdb) [homeassistant.components.influxdb] Cannot connect to InfluxDB due to '[Errno 30] Read-only file system'. Please check that the provided connection details (host, port, etc.) are correct and that your InfluxDB server is running and accessible.

And no data is pushed to the InfluxDB server. ZHA withg NCP and RCP and Thread looks working OK

dimkin-eu commented 9 months ago

@MattWestb try to drop influx container

MattWestb commented 9 months ago

Thanks @dimkin-eu but is was not helping still all InfluxDB sensors is not saved in the bucket.

yiweiyu32 commented 9 months ago

Updated docker packages on x86_64. All working fine. docker-ce version (5:25.0.1-1~debian.12~bookworm)

yiweiyu32 commented 9 months ago

Debian Bookworm here as well.... you guys rock. Saved my install. Thanks!

Same story here. Almost started wiping the system and doing fresh install when found this thread. Downgrade helped well.

One question: why is this bug marked as "unsupported"?

Is supervised installation not one of the official ways to host HA? According to official docs my system is perfectly supported, and I just did my regular system update (what is also my "responsibility" according to the docs).

It's unsupported in the sense that "It's not the hass team's fault. Someone else messed up their updates. Go hound the Moby people for a fix."

fireashes commented 9 months ago

Updated to the following after unhold

sudo apt update

sudo apt list --upgradable

docker-buildx-plugin/bookworm 0.12.1-1~debian.12~bookworm amd64 [upgradable from: 0.11.2-1~debian.12~bookworm]
docker-ce-cli/bookworm 5:25.0.1-1~debian.12~bookworm amd64 [upgradable from: 5:24.0.7-1~debian.12~bookworm]
docker-ce-rootless-extras/bookworm 5:25.0.1-1~debian.12~bookworm amd64 [upgradable from: 5:24.0.7-1~debian.12~bookworm]
docker-ce/bookworm 5:25.0.1-1~debian.12~bookworm amd64 [upgradable from: 5:24.0.7-1~debian.12~bookworm]
docker-compose-plugin/bookworm 2.24.2-1~debian.12~bookworm amd64 [upgradable from: 2.21.0-1~debian.12~bookworm]

sudo apt-mark unhold \
docker-compose-plugin \
docker-ce-cli \
docker-buildx-plugin \
docker-ce \
docker-ce-rootless-extras

apt upgrade -y && apt install -f

Worked flawlessly after reboot of the machine

davecoutts commented 9 months ago

looks like 25.X is braking InfluxDB then all my systems is getting this:
ERROR (influxdb) [homeassistant.components.influxdb] Cannot connect to InfluxDB due to '[Errno 30] Read-only file system'. Please check that the provided connection details (host, port, etc.) are correct and that your InfluxDB server is running and accessible.
And no data is pushed to the InfluxDB server. ZHA withg NCP and RCP and Thread looks working OK

@MattWestb did you manage to resolve the InfluxDB issue?

I have notice the same issue with the same error message. The '[Errno 30] Read-only file system' error message appears to be coming from the a raised OSError here

My InfluxDB instance is on a different server that has not changed, so its something at the HA server end. If I fall back to docker-ce 24.0.7 on the HA server the connection between HA and InfluxDB starts working again and no error is seen in the HA logs.

I wonder if it relates to moby/issues/47158 and the mounting behavior change in v25. The influxdb python client makes use of multiprocessing, which the moby issue states is impacted by the change in permissions of /dev/shm

I'm speculating but its possible other modules that use multiprocessing in HA are impacted by docker-ce v25 and its change in mounting behavior.

MattWestb commented 9 months ago

I was updating 3 test systems and all was doing the same and i was rebootiong one before doing it and it was not having the error. And i was also rolling the production system then i is having more important things things that need the fix on it and loosing all sensor history. The server is one Lubuntu AMD64 running docker containers and is not changed and is not giving anythings in the logs and the container is made of one portainer stack:

services:
  influxdb2:
    container_name: influxdb2
    image: influxdb:latest
    restart: unless-stopped
    ports:
      - '8086:8086'
    environment:
      - TZ=Europe/Vienna
    volumes:
      - /volume1/influxdb2:/var/lib/influxdb2
      - /etc/influxdb2:/etc/influxdb2

And the version is 2.7.3. Reading the code you pointing at its one connection error that HA is logging as read error (its not completely false but better saying cant connection error). So more then likely somthing is going wrong in the network connection and HA is not doing all OK with the in Docker 25.01.

davecoutts commented 9 months ago

I intend to raise a new issue in core for the docker-ce 25.0.x InfluxDB connectivity issue 1 2. This issue has tracked what was a docker problem, not a supervisor problem, for long enough.

gulliver103 commented 9 months ago

All good Debian 11 Raspberry pi 4 Home Assistant observer Supervisor: | Connected Supported: | Supported Healthy: | Healthy

sudo apt update

sudo apt-mark unhold \
docker-compose-plugin \
docker-ce-cli \
docker-buildx-plugin \
docker-ce \
docker-ce-rootless-extras

apt upgrade -y && apt install -f

sudo apt-cache policy docker-ce docker-ce-cli docker-ce-rootless-extras docker-ce: Installed: 5:25.0.1-1~debian.11~bullseye Candidate: 5:25.0.1-1~debian.11~bullseye

MattWestb commented 9 months ago

Thanks @davecoutts !! I was putting core and influxdb in full debug on one test system and nothing more interesting was coming in the HA log. Im very likely going abroad on Friday and being away for one week so cant testing more but then im at home i can doing more testing if needed for fixing the issue.

davecoutts commented 9 months ago

Thanks @MattWestb

For anybody experiencing InfluxDB Integration failures following the update to docker-ce 25.0.1, I have raised an issue in core, core/issues/108810

xdedek commented 9 months ago

Hello at all... I have the same error in my HA on Debian 12 bookworm on a nuc....

In the observer I have this logs error :
Supervisor disconnected

24-01-19 15:27:50 CRITICAL (MainThread) [supervisor.supervisor] Can't setup Supervisor Docker container!
Many addons are out...

I am not a geek of the HA system but I have understand the problem is an update of docker CE....

But I am frenchee and use google to translate.... Could you explain properly to me the commands to pass (under putty ??) for to downgrade docker CE ?

Thanks

I wrote a comment at the end of the post for your case (debian12): https://github.com/home-assistant/supervisor/issues/4827#issuecomment-1902554365

titou4307 commented 9 months ago

@xdedek I have follow all the process and all was OK now

I am waiting the new update docker now

agners commented 9 months ago

This issue was caused by a bug in upstream Docker 25.0.0. As mentioned in https://github.com/home-assistant/supervisor/issues/4827#issuecomment-1907916270, 25.0.1 fixed this problem. I am closing the issue as there is nothing to be done on Supervisor side.

Thanks for reporting and help getting to the bottom of this!

damiano75 commented 9 months ago

Upgrading to Supervisor 2024.01.1 needs to rollback again...

home-assistant / supervisor