home-assistant / supervisor

:house_with_garden: Home Assistant Supervisor
https://home-assistant.io/hassio/
Apache License 2.0
1.79k stars 653 forks source link

Cannot access supervisor "unable to load panel source: /api/hassio/app/entrypoint.js" #1576

Open Falsen opened 4 years ago

Falsen commented 4 years ago

Home Assistant release with the issue: 0.106.5

Operating environment (HassOS/Generic): Raspberry PI, installed with .iso

Supervisor logs:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] udev.sh: executing... 
[15:22:28] INFO: Update udev informations
[cont-init.d] udev.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
20-03-08 15:22:56 INFO (MainThread) [__main__] Initialize Supervisor setup
20-03-08 15:22:57 INFO (SyncWorker_0) [supervisor.docker.supervisor] Attach to Supervisor homeassistant/armv7-hassio-supervisor with version 209
20-03-08 15:22:57 INFO (MainThread) [__main__] Setup Supervisor
20-03-08 15:22:58 INFO (MainThread) [supervisor.utils.gdbus] Connect to dbus: org.freedesktop.systemd1 - /org/freedesktop/systemd1
20-03-08 15:22:58 INFO (MainThread) [supervisor.utils.gdbus] Connect to dbus: org.freedesktop.hostname1 - /org/freedesktop/hostname1
20-03-08 15:22:58 INFO (MainThread) [supervisor.utils.gdbus] Connect to dbus: de.pengutronix.rauc - /
20-03-08 15:22:58 INFO (MainThread) [supervisor.utils.gdbus] Connect to dbus: org.freedesktop.NetworkManager - /org/freedesktop/NetworkManager/DnsManager
20-03-08 15:22:58 INFO (MainThread) [supervisor.host.info] Update local host information
20-03-08 15:22:58 INFO (MainThread) [supervisor.utils.gdbus] Call org.freedesktop.DBus.Properties.GetAll on /org/freedesktop/hostname1
20-03-08 15:22:58 INFO (MainThread) [supervisor.host.services] Update service information
20-03-08 15:22:58 INFO (MainThread) [supervisor.utils.gdbus] Call org.freedesktop.systemd1.Manager.ListUnits on /org/freedesktop/systemd1
20-03-08 15:22:58 INFO (MainThread) [supervisor.host.network] Update local network DNS information
20-03-08 15:22:58 INFO (MainThread) [supervisor.utils.gdbus] Call org.freedesktop.DBus.Properties.GetAll on /org/freedesktop/NetworkManager/DnsManager
20-03-08 15:22:58 INFO (MainThread) [supervisor.host.sound] Update PulseAudio information
20-03-08 15:22:58 INFO (MainThread) [supervisor.host.apparmor] Load AppArmor Profiles: {'hassio-supervisor'}
20-03-08 15:22:58 INFO (MainThread) [supervisor.host.services] Reload local service hassos-apparmor.service
20-03-08 15:22:58 INFO (MainThread) [supervisor.utils.gdbus] Call org.freedesktop.systemd1.Manager.ReloadOrRestartUnit on /org/freedesktop/systemd1
20-03-08 15:22:58 INFO (SyncWorker_2) [supervisor.docker.interface] Attach to homeassistant/armv7-hassio-dns with version 1
20-03-08 15:22:58 INFO (SyncWorker_4) [supervisor.docker.interface] Attach to homeassistant/armv7-hassio-audio with version 8
20-03-08 15:22:58 INFO (MainThread) [supervisor.dns] Start CoreDNS plugin
20-03-08 15:22:58 INFO (MainThread) [supervisor.misc.forwarder] Start DNS port forwarding to 172.30.32.3
20-03-08 15:22:59 INFO (MainThread) [supervisor.audio] Start Audio plugin
20-03-08 15:22:59 INFO (SyncWorker_5) [supervisor.docker.interface] Clean hassio_dns application
20-03-08 15:22:59 INFO (SyncWorker_4) [supervisor.docker.interface] Clean hassio_audio application
20-03-08 15:23:03 INFO (SyncWorker_5) [supervisor.docker.dns] Start DNS homeassistant/armv7-hassio-dns with version 1 - 172.30.32.3
20-03-08 15:23:04 INFO (SyncWorker_4) [supervisor.docker.audio] Start Audio homeassistant/armv7-hassio-audio with version 8 - 172.30.32.4
20-03-08 15:23:04 INFO (SyncWorker_9) [supervisor.docker.interface] Attach to homeassistant/raspberrypi2-homeassistant with version 0.106.5
20-03-08 15:23:04 INFO (MainThread) [supervisor.utils.gdbus] Call org.freedesktop.DBus.Properties.GetAll on /
20-03-08 15:23:04 INFO (MainThread) [supervisor.hassos] Detect HassOS 3.12 / BootSlot B
20-03-08 15:23:05 INFO (SyncWorker_10) [supervisor.docker.hassos_cli] Found HassOS CLI homeassistant/armv7-hassio-cli with version 21
20-03-08 15:23:07 INFO (MainThread) [supervisor.store.git] Load add-on /data/addons/core repository
20-03-08 15:23:07 INFO (MainThread) [supervisor.store] Load add-ons from store: 20 all - 20 new - 0 remove
20-03-08 15:23:07 INFO (MainThread) [supervisor.addons] Found 3 installed add-ons
20-03-08 15:23:07 INFO (SyncWorker_8) [supervisor.docker.interface] Attach to homeassistant/armv7-addon-configurator with version 4.7
20-03-08 15:23:07 INFO (SyncWorker_3) [supervisor.docker.interface] Attach to homeassistant/armv7-addon-ssh with version 7.1
20-03-08 15:23:07 INFO (SyncWorker_4) [supervisor.docker.interface] Attach to homeassistant/armv7-addon-letsencrypt with version 4.7
20-03-08 15:23:08 INFO (MainThread) [supervisor.updater] Fetch update data from https://version.home-assistant.io/stable.json
20-03-08 15:23:12 INFO (MainThread) [supervisor.snapshots] Found 3 snapshot files
20-03-08 15:23:12 INFO (MainThread) [supervisor.discovery] Load 0 messages
20-03-08 15:23:12 INFO (MainThread) [supervisor.ingress] Load 3 ingress session
20-03-08 15:23:12 INFO (MainThread) [supervisor.secrets] Load Home Assistant secrets: 1
20-03-08 15:23:12 INFO (MainThread) [__main__] Run Supervisor
20-03-08 15:23:12 INFO (MainThread) [supervisor.api] Start API on 172.30.32.2
20-03-08 15:23:12 INFO (MainThread) [supervisor.utils.gdbus] Call de.pengutronix.rauc.Installer.Mark on /
20-03-08 15:23:12 INFO (MainThread) [supervisor.hassos] Rauc: B - marked slot kernel.1 as good
20-03-08 15:23:12 INFO (MainThread) [supervisor.addons] Phase 'initialize' start 0 add-ons
20-03-08 15:23:12 INFO (MainThread) [supervisor.addons] Phase 'system' start 0 add-ons
20-03-08 15:23:12 INFO (MainThread) [supervisor.addons] Phase 'services' start 0 add-ons
20-03-08 15:23:12 INFO (SyncWorker_10) [supervisor.docker.interface] Start homeassistant/raspberrypi2-homeassistant
20-03-08 15:24:19 ERROR (MainThread) [supervisor.api.security] Invalid token for access /homeassistant/options
20-03-08 15:24:19 ERROR (MainThread) [supervisor.api.security] Invalid token for access /supervisor/options
20-03-08 15:24:19 ERROR (MainThread) [supervisor.api.security] Invalid token for access /homeassistant/info
20-03-08 15:24:19 ERROR (MainThread) [supervisor.api.security] Invalid token for access /ingress/panels
20-03-08 15:24:20 ERROR (MainThread) [supervisor.api.security] Invalid token for access /discovery
20-03-08 15:24:24 INFO (MainThread) [supervisor.homeassistant] Detect a running Home Assistant instance
20-03-08 15:24:24 INFO (MainThread) [supervisor.addons] Phase 'application' start 1 add-ons
20-03-08 15:24:24 INFO (SyncWorker_11) [supervisor.docker.interface] Clean addon_core_configurator application
20-03-08 15:24:28 INFO (SyncWorker_11) [supervisor.docker.addon] Start Docker add-on homeassistant/armv7-addon-configurator with version 4.7
20-03-08 15:24:33 ERROR (MainThread) [supervisor.api.security] Invalid token for access /app/entrypoint.js
20-03-08 15:24:33 INFO (MainThread) [supervisor.tasks] All core tasks are scheduled
20-03-08 15:24:33 INFO (MainThread) [supervisor.hwmon] Started Supervisor hardware monitor
20-03-08 15:24:33 INFO (MainThread) [supervisor.core] Supervisor is up and running
20-03-08 15:24:33 INFO (MainThread) [supervisor.host.info] Update local host information
20-03-08 15:24:33 INFO (MainThread) [supervisor.utils.gdbus] Call org.freedesktop.DBus.Properties.GetAll on /org/freedesktop/hostname1
20-03-08 15:24:33 INFO (MainThread) [supervisor.updater] Fetch update data from https://version.home-assistant.io/stable.json
20-03-08 15:24:34 INFO (MainThread) [supervisor.host.services] Update service information
20-03-08 15:24:34 INFO (MainThread) [supervisor.utils.gdbus] Call org.freedesktop.systemd1.Manager.ListUnits on /org/freedesktop/systemd1
20-03-08 15:24:34 INFO (MainThread) [supervisor.host.network] Update local network DNS information
20-03-08 15:24:34 INFO (MainThread) [supervisor.utils.gdbus] Call org.freedesktop.DBus.Properties.GetAll on /org/freedesktop/NetworkManager/DnsManager
20-03-08 15:24:35 INFO (MainThread) [supervisor.host.sound] Update PulseAudio information
20-03-08 15:36:56 ERROR (MainThread) [supervisor.api.security] Invalid token for access /app/entrypoint.js
20-03-08 16:00:21 ERROR (MainThread) [supervisor.api.security] Invalid token for access /app/entrypoint.js
20-03-08 16:03:50 ERROR (MainThread) [supervisor.api.security] Invalid token for access /app/entrypoint.js

Description of problem: The problem first occurred after updating, I had a quite old version before. From mid-January or something. After updating the supervisor panel was no longer accessible and I got the error "Unable to load panel source: /api/hassio/app/entrypoint.js". Naturally, I restarted Home Assistant & the PI a few times, and suddenly it started working again. Later I was working on some other things with my configuration which required me to restart Hass, at first it was working well. However, after a few restarts, the issue occurred again.

I am very new to Home Assistant and I just recently figured out how to gain SSH access, so please tell me if there is something I can try.

phunkybeam commented 4 years ago

Since I have installed the update 0.106.4 on HassOS, this error also appears with me.

So I cannot access the supervisor anymore.

“Unable to load the panel source: /api/hassio/app/entrypoint.js.”

What do I do now if I don't have SSH access?

Node Red is not installed.

Protokolldetails ( ERROR )

Logger: homeassistant.components.hassio.http Integration: Hass.io (documentation, issues) First occured: 6. März 2020, 22:06:36 (18 occurences) Last logged: 19:09:05

Client error on api app/entrypoint.js request Cannot connect to host 172.30.32.2:80 ssl:None [Connect call failed ('172.30.32.2', 80)]

https://community.home-assistant.io/t/cant-enter-in-supervisor-unable-to-load-the-panel-source-0-106-4/176988

https://community-home-assistant-assets.s3.dualstack.us-west-2.amazonaws.com/original/3X/e/3/e33406e5c1fa4f783a82e82ac5e03e272d536ed4.jpeg

phunkybeam commented 4 years ago

All good again, it was probably HACS after an update and a HardReset it runs again.

rsnodgrass commented 4 years ago

This specific error is always the bane of me. I manage three different HA installations and inevitably over the last 18 months this starts happening every few months on one of them. A painful full reinstall and re-setup is required.

It sure would be nice for some way to "reinstall" or reset the supervisor FROM homeassistant core so this is resolved without having to muck around with manually updating things on the command line, or performing a full reinstall.

chris669 commented 4 years ago

Similar problem for me since latest supervisor update. Running latest HA on RPI3B+ Screenshot_20200329-220947 Tried to relaunch HA didn't helped

colinfrei commented 4 years ago

This issue started out of the blue for me last night as well.

BasTijs commented 4 years ago

Same here, I've not updated anything and now I get the same error. I also see this in the logs Client error on api app/entrypoint.js request Cannot connect to host 172.30.32.2:80 ssl:None [Connect call failed ('172.30.32.2', 80)]

After reboot the error is gone but also my hassio/supervior dashboard is empty, only showing the core and OS updates

colinfrei commented 4 years ago

I could resolve this through updating HACS and restarting a few times - not sure what helped exactly.

chris669 commented 4 years ago

Can someone please gives me the latest command line to reboot Supervisor, as it changes all the time...

BasTijs commented 4 years ago

I could resolve this through updating HACS and restarting a few times - not sure what helped exactly.

For me this went into a 500 error and after hard-reboot I cant load the HA webinterface anymore.... anyone know how to remove or update HACS through command line or anything I can try that may solve this? I can access it through samba and SSH.

chris669 commented 4 years ago

My problem is when I try to reboot with ha supervisor restart I get a "connection refused" ?? My Google 2 factors is enabled, may be this the cause?

marcozoppy commented 4 years ago

Some thing to me. I noticed the CPU around 90% all the time. And no access to supervisor.

chris669 commented 4 years ago

@pvizeli, what's your suggestion as command lines are refused (connection) and web interface not available? Dirty (unplug) reboot??

monpelaud commented 4 years ago

Hi, Same issue as @chris669.

Can't access to supervisor from web interface: Unable to load the panel source: /api/hassio/app/entrypoint.js.

From the CLI I get a connection refused error message:

ha supervisor info or ha supervisor reload or ha supervisor update or ha supervisor logs

Get http://supervisor/supervisor/info: dial tcp 172.30.32.2:80: connect: connection refused

No reboot, no dirty unplug. I have not yet tried to reboot de raspberry pi3 because I still have access by ssh. I'm afraid of losing ssh access after the reboot. I use hassio with Home Assistant 0.107.7.

Best regards

chris669 commented 4 years ago

Ok, as no other solutions, I tried the dirty reboot, unplugging my RPI3 and as I was very lucky, it worked, but I hate doing this. So good luck!

thom13 commented 4 years ago

Ok, as no other solutions, I tried the dirty reboot, unplugging my RPI3 and as I was very lucky, it worked, but I hate doing this. So good luck!

Same thing for me Hard reboot ! I uninstall HACS... maybe it's the problem

JayOne73 commented 4 years ago

Also the same problem here. Rebooting doesn't seem the solution so far. After reboot I can ssh, but the frontend doesn't come alive. HA 0.106.5, on a RPI3

BasTijs commented 4 years ago

Also the same problem here. Rebooting doesn't seem the solution so far. After reboot I can ssh, but the frontend doesn't come alive. HA 0.106.5, on a RPI3

I will reinstall soon because of the problems this introduced, but I managed to get in by doing a 'hassio ha check" and fixed the new config errors due to the auto update. So I cleared the config file and custom_components which allowed to make a snapshots (save your databases etc..) once I got in.

JayOne73 commented 4 years ago

Also the same problem here. Rebooting doesn't seem the solution so far. After reboot I can ssh, but the frontend doesn't come alive. HA 0.106.5, on a RPI3

I will reinstall soon because of the problems this introduced, but I managed to get in by doing a 'hassio ha check" and fixed the new config errors due to the auto update. So I cleared the config file and custom_components which allowed to make a snapshots (save your databases etc..) once I got in.

Well, that didn't the trick for me. So I have installed HA from scratch with a snapshot from 4 days ago. Only excluded the Googledrive backup addon, because that addon stopped working a few days ago. For now the system is up and running again with supervisor 214.

Really pisses me off that you can't control the updates of the supervisor and breaks a running instance after an automatic update.

floari commented 4 years ago

Same problem here. Also rerunning the install script did not solve the issue.

dereitz commented 4 years ago

Same problem here. I had been having intermittent issues for a while that might have been related to running watchtower (which kept trying to update the supervisor). I have completely removed watchtower, and now I get this error 100% of the time. Would appreciate if anyone could help!!

dereitz commented 4 years ago

Just to update. I completely stopped all services, all docker processes, and removed all my docker containers and images that were associated with the HA installation on my system, but did NOT remove anything from /usr/share/hassio where the config/data is stored. I then installed from scratch using the install script, and my system is once again working. It makes no sense to me why a reinstall would work, but sure enough it did for me. Performing the reinstall was far easier than trying to debug the issue for me!

ajayjohn commented 4 years ago

@dereitz It works everytime for me too. Basically, if the homeassitant and supervisor images are pulled fresh from the repos for a reinstall, everything fixes itself and starts working. But, a few minutes or sometimes hours later, the same issue resurfaces.

I'll add some logs here shortly, but from what I noticed, there seems to be some DNS related lines that come up before this issue comes up again. It usually happens when it's trying to load some add-on.

r100gs commented 4 years ago

I face the same problem. ATM I dont have the time to digg deeper into, but maybe tomorrow I will try to reinstall all docker thing. My Installation ist on homeassistan 107.7

Its anoying!

oooseun commented 4 years ago

Just want to comment that i have the same issue every couple of months (or weeks?) AND IT DRIVES ME CRAZY

sailhobie commented 4 years ago

same issue here

Kanishkaz commented 4 years ago

I constantly get this same issue too.

devtech8 commented 4 years ago

I too am having the same issue. Mine runs via Docker and portainer shows hassio supervisor updated April 5th sometime. Trued restarting container, but still face the same issue. Anyone have luck on this?

chris669 commented 4 years ago

A simple message like " ok, we're working on it" would be appreciated...

Justalurker1213 commented 4 years ago

I’m having this same issue but it’s happened every night the past 3 nights. The only way I’ve been able to fix it is to stop the supervisor services, remove all supervisor containers along with the home assistant container, rerun the setup script, and reboot. This gets everything working again and all is well all day but sometime around midnight it breaks again. Very frustrating because when it breaks all of my addons stop working which means no node red, which means no security system while we sleep. Is there any way to figure out what exactly is going on to make this happen?

I also have watchtower running. Could the issue be that it's updating Supervisor around midnight? I've disabled the container and will see if it breaks again this evening.

dereitz commented 4 years ago

Is anyone running Watchtower? If so, check the Watchtower logs to see if it is attempting to update your HASSIO supervisor container. You may also want to keep an eye on the uptime of the HASSIO supervisor container.

In my case, Watchtower was attempting to update the HASSIO Supervisor container, and it was restarting the container in order to attempt to do so. Again, I have no idea why this would "corrupt" the install, but hoping we can find a pattern in order to assist the developers in determining the root cause.

ludeeus commented 4 years ago

Disable watchtower for any container that is managed by HA (supervisor).

pvizeli commented 4 years ago

Please read all: https://github.com/home-assistant/supervised-installer#warning

Use Home Assistant only on a dedicated machine otherwise you run into issue. If you are a linux or docker high level admin, you can fix the issues which such system will give, otherwise I strict recommend to not do that.

Also don't block ntp or dns on your network if you are not 100% sure what you do and how you can fix software they did not like this. (dns is not so a issue as like ntp). After that you are happy Home Assistant user without issue (mostly) - the internal docker database can going corrupt with power off, but that is not often happen or can fix with ha su repair.

colinfrei commented 4 years ago

FWIW, I'm running HA installed using the then hass.io image, so nothing else on the machine. I also don't have any network blocks.

monpelaud commented 4 years ago

Hi, I'm using a dedicated Pi3 rapsberry with a hassio image. No blocks on my network, I use this configuration for several months without any problem. No change on my network. I use few addons and only apply official hassio updates. Best regards

Falsen commented 4 years ago

Hi, I'm using a dedicated Pi3 rapsberry with a hassio image. No blocks on my network, I use this configuration for several months without any problem. No change on my network. I use few addons and only apply official hassio updates. Best regards

That's my setup too and it fixed itself after a few restarts and I've never had the same problem again

alexsands commented 4 years ago

Going off @scsmartt and @dereitz 's comments, I also found my issue to be watchtower related. I've had it for several months now but wasn't a problem until the past week. I think watchtower tried to auto update hassio_supervisor at some point in the last few days (my cron is every SAT, midnight) and it broke.

For others who may also have a similar setup of Ubuntu 18.04, docker, portainer, watchtower, and install via supervised-installer, I fixed this issue by:

  1. Remove watchtower from docker-compose.yml. Kill and remove the container in Portainer. Then run docker-compose up -d --remove-orphans to save. Watchtower should now be gone.
  2. Prepare to remove docker containers by stopping/disabling/removing services. This comment explains how to very well.
  3. Kill and remove the following docker containers via Portainer or cli: hassio_supervisor, homeassistant, addon_*. Make sure they didn't restart automatically (if they did, you need to do step 2 again).
  4. Run supervised-installer as root (sudo su) with (+ any optional flags, see install instructions):
    curl -sL https://raw.githubusercontent.com/home-assistant/supervised-installer/master/installer.sh | bash -s
  5. Check Portainer to see that your hassio_supervisor container is running again. If homeassistant didn't start, run ha core start (reference).
  6. Wait for everything to boot up (may take 1-2 min).
  7. Visit your HA page. Now you should have access to the Supervisor tab and can start your other add-ons.

Hopefully this helps! Spent a good amount of time trying to figure this out myself, since I'm relatively new to HA.

Kanishkaz commented 4 years ago

Going off @scsmartt and @dereitz 's comments, I also found my issue to be watchtower related. I've had it for several months now but wasn't a problem until the past week. I think watchtower tried to auto update hassio_supervisor at some point in the last few days (my cron is every SAT, midnight) and it broke.

For others who may also have a similar setup of Ubuntu 18.04, docker, portainer, watchtower, and install via supervised-installer, I fixed this issue by:

  1. Remove watchtower from docker-compose.yml. Kill and remove the container in Portainer. Then run docker-compose up -d --remove-orphans to save. Watchtower should now be gone.
  2. Prepare to remove docker containers by stopping/disabling/removing services. This comment explains how to very well.
  3. Kill and remove the following docker containers via Portainer or cli: hassio_supervisor, homeassistant, addon_*. Make sure they didn't restart automatically (if they did, you need to do step 2 again).
  4. Run supervised-installer as root (sudo su) with (+ any optional flags, see install instructions):
curl -sL https://raw.githubusercontent.com/home-assistant/supervised-installer/master/installer.sh | bash -s
  1. Check Portainer to see that your hassio_supervisor container is running again. If homeassistant didn't start, run ha core start (reference).
  2. Wait for everything to boot up (may take 1-2 min).
  3. Visit your HA page. Now you should have access to the Supervisor tab and can start your other add-ons.

Hopefully this helps! Spent a good amount of time trying to figure this out myself, since I'm relatively new to HA.

This worked for me too. However, I kept watchtower by enabling --label-enable to ensure it only monitor containers I pick rather than all containers. Just have to wait and see if this sticks now.

monpelaud commented 4 years ago

Hi, Unfortunately I don't use watchtower! I think watchtower malfunction is a consequence, not the root cause.

Best regards

Justalurker1213 commented 4 years ago

Going off @scsmartt and @dereitz 's comments, I also found my issue to be watchtower related. I've had it for several months now but wasn't a problem until the past week. I think watchtower tried to auto update hassio_supervisor at some point in the last few days (my cron is every SAT, midnight) and it broke.

This works to be sure and thanks for the info. The problem is I have to do it every morning. 😰 It did it to me again this morning even with Watchtower stopped. One thing I noticed in Portainer after rebooting is there was a new hassio multicast container that was stopped and wouldn't start. I've never seen it before. Portainer says it was created around midnight last night while I was counting sheep and definitely not tinkering with HASS. I ran "ha su repair", it completed successfully, and the multicast container was gone. Supervisor is still broken so I'll have to go through the rigamarole again but maybe that's a clue as to why it's breaking in the first place?

@monpelaud is this occurring on a nightly basis for you as well?

JayOne73 commented 4 years ago

Hi, Unfortunately I don't use watchtower! I think watchtower malfunction is a consequence, not the root cause.

Best regards

I also don't use Watchtower or any other docker management and I run just the image version (RPI3) with approx. ten addons. So I think that the problem is more within the supervisor, because my problem occurred after the 214 upgrade.

monpelaud commented 4 years ago

Going off @scsmartt and @dereitz 's comments, I also found my issue to be watchtower related. I've had it for several months now but wasn't a problem until the past week. I think watchtower tried to auto update hassio_supervisor at some point in the last few days (my cron is every SAT, midnight) and it broke.

This works to be sure and thanks for the info. The problem is I have to do it every morning. 😰 It did it to me again this morning even with Watchtower stopped. One thing I noticed in Portainer after rebooting is there was a new hassio multicast container that was stopped and wouldn't start. I've never seen it before. Portainer says it was created around midnight last night while I was counting sheep and definitely not tinkering with HASS. I ran "ha su repair", it completed successfully, and the multicast container was gone. Supervisor is still broken so I'll have to go through the rigamarole again but maybe that's a clue as to why it's breaking in the first place?

@monpelaud is this occurring on a nightly basis for you as well?

Not every night, it's random.

Justalurker1213 commented 4 years ago

My setup is working ok this morning. Turns out I had added labels for watchtower to my containers but had not actually enabled labels in watchtower so I'm guessing it was still updating and breaking some part of HASS. I enabled labels yesterday and this morning everything is good. I'm not sure if it's coincidence or not.

I did notice quite a few of these in my supervisor logs:

20-04-07 23:33:53 ERROR (MainThread) [supervisor.api.proxy] Client error on WebSocket API Cannot connect to host 172.30.32.1:8123 ssl:False [Connection refused].

I'm not sure what that IP is. It doesn't exist in my docker instance as far as I can tell but it's the same subnet as the supervisor container. They stopped after 11:34pm, right around the time it's been breaking the past few nights.

dereitz commented 4 years ago

My setup is working ok this morning. Turns out I had added labels for watchtower to my containers but had not actually enabled labels in watchtower so I'm guessing it was still updating and breaking some part of HASS. I enabled labels yesterday and this morning everything is good. I'm not sure if it's coincidence or not.

I did notice quite a few of these in my supervisor logs:

20-04-07 23:33:53 ERROR (MainThread) [supervisor.api.proxy] Client error on WebSocket API Cannot connect to host 172.30.32.1:8123 ssl:False [Connection refused].

I'm not sure what that IP is. It doesn't exist in my docker instance as far as I can tell but it's the same subnet as the supervisor container. They stopped after 11:34pm, right around the time it's been breaking the past few nights.

Can you please share the labels you are using? If nothing else, I think it would be good to establish a "best practice" for the use of Watchtower in combination with HASS. Thanks in advance!

darrylb83 commented 4 years ago

The below worked for me, I am going to try to update to the latest. Upon re-running the installer, it worked and all my config was there. Going to experiment and continue building comfort in stability. Leaving watchtower off, as I dont need it any longer. :)

Going off @scsmartt and @dereitz 's comments, I also found my issue to be watchtower related. I've had it for several months now but wasn't a problem until the past week. I think watchtower tried to auto update hassio_supervisor at some point in the last few days (my cron is every SAT, midnight) and it broke.

For others who may also have a similar setup of Ubuntu 18.04, docker, portainer, watchtower, and install via supervised-installer, I fixed this issue by:

  1. Remove watchtower from docker-compose.yml. Kill and remove the container in Portainer. Then run docker-compose up -d --remove-orphans to save. Watchtower should now be gone.
  2. Prepare to remove docker containers by stopping/disabling/removing services. This comment explains how to very well.
  3. Kill and remove the following docker containers via Portainer or cli: hassio_supervisor, homeassistant, addon_*. Make sure they didn't restart automatically (if they did, you need to do step 2 again).
  4. Run supervised-installer as root (sudo su) with (+ any optional flags, see install instructions):
curl -sL https://raw.githubusercontent.com/home-assistant/supervised-installer/master/installer.sh | bash -s
  1. Check Portainer to see that your hassio_supervisor container is running again. If homeassistant didn't start, run ha core start (reference).
  2. Wait for everything to boot up (may take 1-2 min).
  3. Visit your HA page. Now you should have access to the Supervisor tab and can start your other add-ons.

Hopefully this helps! Spent a good amount of time trying to figure this out myself, since I'm relatively new to HA.

bckthomas commented 4 years ago

Fresh new install of hassio yesterday, everything worked fine. After hassio update to 214, I get the error. I use hassio docker image on unraid

Justalurker1213 commented 4 years ago

Can you please share the labels you are using? If nothing else, I think it would be good to establish a "best practice" for the use of Watchtower in combination with HASS. Thanks in advance!

I used this for reference: https://containrrr.github.io/watchtower/container-selection/

Basically, just enable labels on the WT container and then add the label to each container you want it to update and it will leave all others alone.

Fresh new install of hassio yesterday, everything worked fine. After hassio update to 214, I get the error. I use hassio docker image on unraid

Do you have Watchtower running? My supervisor is on 214 as well and running fine since keeping WT from updating anything to do with HASS. I would run “ha su repair” on your server and if that gets it working again, update everything within HASS. The latest install script should install 214 by default however.

bckthomas commented 4 years ago

Do you have Watchtower running? My supervisor is on 214 as well and running fine since keeping WT from updating anything to do with HASS. I would run “ha su repair” on your server and if that gets it working again, update everything within HASS. The latest install script should install 214 by default however.

I don't use watchtower. Please, can you tell me how can I execute "ha su repair" in homeassistant/amd64-hassio-supervisor docker image ?

ludeeus commented 4 years ago

You do it in the hassio_cli container

bckthomas commented 4 years ago

Thanks @ludeeus But unfortunately, no changes. Here is the docker command for hassio

docker run -d --name='hassio_supervisor' --net='bridge' --privileged=true -e TZ="Europe/Paris" -e HOST_OS="Unraid" -e 'SUPERVISOR_SHARE'='/mnt/user/appdata/hassio/' -e 'SUPERVISOR_NAME'='hassio_supervisor' -e 'HOMEASSISTANT_REPOSITORY'='homeassistant/intel-nuc-homeassistant' -v '/mnt/user/appdata/hassio/':'/data':'rw' -v '/var/run/docker.sock':'/var/run/docker.sock':'rw' -v '/var/run/dbus':'/var/run/dbus':'rw' --security-opt apparmor:unconfined --security-opt seccomp=unconfined 'homeassistant/amd64-hassio-supervisor'

And the logs for this command 20-04-11 12:15:32 INFO (MainThread) [__main__] Initialize Supervisor setup 20-04-11 12:15:32 INFO (SyncWorker_0) [supervisor.docker.supervisor] Attach to Supervisor homeassistant/amd64-hassio-supervisor with version 214 20-04-11 12:15:32 INFO (SyncWorker_0) [supervisor.docker.supervisor] Connect Supervisor to hassio Network 20-04-11 12:15:33 INFO (MainThread) [__main__] Setup Supervisor 20-04-11 12:15:33 WARNING (MainThread) [supervisor.dbus.systemd] No systemd support on the host. Host control has been disabled. 20-04-11 12:15:33 WARNING (MainThread) [supervisor.dbus.hostname] No hostname support on the host. Hostname functions have been disabled. 20-04-11 12:15:33 WARNING (MainThread) [supervisor.dbus.rauc] Host has no rauc support. OTA updates have been disabled. 20-04-11 12:15:33 WARNING (MainThread) [supervisor.dbus.nmi_dns] No DnsManager support on the host. Local DNS functions have been disabled. 20-04-11 12:15:33 INFO (MainThread) [supervisor.host.sound] Update PulseAudio information 20-04-11 12:15:33 INFO (MainThread) [supervisor.host.apparmor] Load AppArmor Profiles: {'hassio-supervisor'} 20-04-11 12:15:33 INFO (MainThread) [supervisor.host.apparmor] AppArmor is not enabled on host 20-04-11 12:15:33 INFO (SyncWorker_2) [supervisor.docker.interface] Attach to homeassistant/amd64-hassio-dns with version 9 20-04-11 12:15:33 INFO (SyncWorker_0) [supervisor.docker.interface] Attach to homeassistant/amd64-hassio-audio with version 14 20-04-11 12:15:33 INFO (SyncWorker_5) [supervisor.docker.interface] Attach to homeassistant/amd64-hassio-cli with version 25 20-04-11 12:15:33 INFO (MainThread) [supervisor.misc.forwarder] Start DNS port forwarding to 172.30.32.3 20-04-11 12:15:33 INFO (SyncWorker_0) [supervisor.docker.interface] Restart homeassistant/amd64-hassio-dns 20-04-11 12:15:38 INFO (MainThread) [supervisor.dns] Updated /etc/resolv.conf 20-04-11 12:15:38 INFO (SyncWorker_10) [supervisor.docker.interface] Attach to homeassistant/intel-nuc-homeassistant with version 0.108.3 20-04-11 12:15:38 INFO (MainThread) [supervisor.store.git] Load add-on /data/addons/git/a0d7b954 repository 20-04-11 12:15:38 INFO (MainThread) [supervisor.store.git] Load add-on /data/addons/core repository 20-04-11 12:15:38 INFO (MainThread) [supervisor.dns] Updated /etc/resolv.conf 20-04-11 12:15:38 INFO (SyncWorker_10) [supervisor.docker.interface] Attach to homeassistant/intel-nuc-homeassistant with version 0.108.3 20-04-11 12:15:38 INFO (MainThread) [supervisor.store.git] Load add-on /data/addons/git/a0d7b954 repository 20-04-11 12:15:38 INFO (MainThread) [supervisor.store.git] Load add-on /data/addons/core repository 20-04-11 12:15:38 INFO (MainThread) [supervisor.store] Load add-ons from store: 65 all - 65 new - 0 remove 20-04-11 12:15:38 INFO (MainThread) [supervisor.addons] Found 2 installed add-ons 20-04-11 12:15:38 INFO (SyncWorker_8) [supervisor.docker.interface] Attach to homeassistant/amd64-addon-samba with version 9.0 20-04-11 12:15:38 INFO (SyncWorker_14) [supervisor.docker.interface] Attach to hassioaddons/vscode-amd64 with version 2.1.0 20-04-11 12:15:38 INFO (MainThread) [supervisor.updater] Fetch update data from https://version.home-assistant.io/stable.json 20-04-11 12:15:39 INFO (MainThread) [supervisor.snapshots] Found 2 snapshot files 20-04-11 12:15:39 INFO (MainThread) [supervisor.discovery] Load 0 messages 20-04-11 12:15:39 INFO (MainThread) [supervisor.ingress] Load 0 ingress session 20-04-11 12:15:39 INFO (MainThread) [supervisor.secrets] Load Home Assistant secrets: 7 20-04-11 12:15:39 INFO (MainThread) [__main__] Run Supervisor 20-04-11 12:15:39 INFO (MainThread) [supervisor.api] Start API on 172.30.32.2 20-04-11 12:15:39 INFO (MainThread) [supervisor.addons] Phase 'initialize' start 0 add-ons 20-04-11 12:15:39 INFO (MainThread) [supervisor.core] Supervisor reboot detected 20-04-11 12:15:39 INFO (MainThread) [supervisor.tasks] All core tasks are scheduled 20-04-11 12:15:39 INFO (MainThread) [supervisor.hwmon] Started Supervisor hardware monitor 20-04-11 12:15:39 INFO (MainThread) [supervisor.core] Supervisor is up and running

And the logs after the "ha su repair" command [32m20-04-11 12:16:33 INFO (MainThread) [supervisor.core] Start repairing of Supervisor Environment 20-04-11 12:16:33 INFO (SyncWorker_6) [supervisor.docker] Prune stale containers 20-04-11 12:16:33 INFO (SyncWorker_6) [supervisor.docker] Prune stale images 20-04-11 12:16:33 INFO (SyncWorker_6) [supervisor.docker] Prune stale builds 20-04-11 12:16:33 INFO (SyncWorker_6) [supervisor.docker] Prune stale volumes 20-04-11 12:16:33 INFO (SyncWorker_6) [supervisor.docker] Prune stale networks 20-04-11 12:16:33 INFO (MainThread) [supervisor.cli] Repair HA cli 25 20-04-11 12:16:33 INFO (SyncWorker_19) [supervisor.docker.interface] Pull image homeassistant/amd64-hassio-cli tag 25. 20-04-11 12:16:35 INFO (SyncWorker_19) [supervisor.docker.interface] Tag image homeassistant/amd64-hassio-cli with version 25 as latest 20-04-11 12:16:35 INFO (MainThread) [supervisor.addons] Found 0 add-ons to repair 20-04-11 12:16:35 INFO (MainThread) [supervisor.supervisor] Repair Supervisor 214 20-04-11 12:16:35 INFO (MainThread) [supervisor.core] Finished repairing of Supervisor Environment

pvizeli commented 4 years ago

@bckthomas You can't just run the Supervisor in a container. This will never work correct.