home-assistant / supervisor

:house_with_garden: Home Assistant Supervisor
https://home-assistant.io/hassio/
Apache License 2.0
1.75k stars 640 forks source link

hassio_supervisor 187 appears to crash (Generic install on Ubuntu 18.04) #1292

Closed skawt-tee closed 5 years ago

skawt-tee commented 5 years ago

Home Assistant release with the issue: 0.98.5

Operating environment (HassOS/Generic): Generic hassio install on Ubuntu 18.04

Supervisor logs: 19-09-14 21:08:56 INFO (MainThread) [main] Initialize Hass.io setup 19-09-14 21:08:56 INFO (SyncWorker_0) [hassio.docker.supervisor] Attach to Supervisor homeassistant/amd64-hassio-supervisor with version 187 19-09-14 21:08:56 INFO (SyncWorker_0) [hassio.docker.supervisor] Connect Supervisor to Hass.io Network 19-09-14 21:08:57 INFO (MainThread) [main] Setup HassIO 19-09-14 21:08:57 INFO (MainThread) [hassio.utils.gdbus] Connect to dbus: org.freedesktop.systemd1 - /org/freedesktop/systemd1 19-09-14 21:08:57 INFO (MainThread) [hassio.utils.gdbus] Connect to dbus: org.freedesktop.hostname1 - /org/freedesktop/hostname1 19-09-14 21:08:57 WARNING (MainThread) [hassio.dbus.rauc] Host has no rauc support. OTA updates have been disabled. 19-09-14 21:08:57 INFO (MainThread) [hassio.utils.gdbus] Connect to dbus: org.freedesktop.NetworkManager - /org/freedesktop/NetworkManager/DnsManager 19-09-14 21:08:57 INFO (MainThread) [hassio.host.info] Update local host information 19-09-14 21:08:57 INFO (MainThread) [hassio.utils.gdbus] Call org.freedesktop.DBus.Properties.GetAll on /org/freedesktop/hostname1 19-09-14 21:08:57 INFO (MainThread) [hassio.host.services] Update service information 19-09-14 21:08:57 INFO (MainThread) [hassio.utils.gdbus] Call org.freedesktop.systemd1.Manager.ListUnits on /org/freedesktop/systemd1 19-09-14 21:08:57 INFO (MainThread) [hassio.host.network] Update local network DNS information 19-09-14 21:08:57 INFO (MainThread) [hassio.utils.gdbus] Call org.freedesktop.DBus.Properties.GetAll on /org/freedesktop/NetworkManager/DnsManager 19-09-14 21:08:57 INFO (MainThread) [hassio.host.apparmor] Load AppArmor Profiles: {'hassio-supervisor'} 19-09-14 21:08:57 INFO (MainThread) [hassio.host.services] Reload local service hassio-apparmor.service 19-09-14 21:08:57 INFO (MainThread) [hassio.utils.gdbus] Call org.freedesktop.systemd1.Manager.ReloadOrRestartUnit on /org/freedesktop/systemd1 19-09-14 21:08:57 INFO (SyncWorker_2) [hassio.docker.interface] Attach to homeassistant/amd64-hassio-dns with version 1 19-09-14 21:08:57 INFO (MainThread) [hassio.dns] Update resolv from Supervisor 19-09-14 21:08:57 WARNING (MainThread) [hassio.dns] Can't write local resolv: [Errno 13] Permission denied: '/etc/resolv.conf' 19-09-14 21:08:57 INFO (MainThread) [hassio.misc.forwarder] Start DNS port forwarding to 172.30.32.3 19-09-14 21:08:57 INFO (SyncWorker_4) [hassio.docker.interface] Stop hassio_dns application 19-09-14 21:08:59 INFO (SyncWorker_4) [hassio.docker.interface] Clean hassio_dns application 19-09-14 21:08:59 INFO (MainThread) [hassio.dns] Start CoreDNS plugin 19-09-14 21:09:01 INFO (SyncWorker_0) [hassio.docker.dns] Start DNS homeassistant/amd64-hassio-dns with version 1 19-09-14 21:09:01 INFO (SyncWorker_6) [hassio.docker.interface] Attach to homeassistant/qemux86-64-homeassistant with version 0.98.5 19-09-14 21:09:01 INFO (MainThread) [hassio.store.git] Load add-on /data/addons/git/a0d7b954 repository 19-09-14 21:09:01 INFO (MainThread) [hassio.store.git] Load add-on /data/addons/git/fe41fc28 repository 19-09-14 21:09:01 INFO (MainThread) [hassio.store.git] Load add-on /data/addons/core repository 19-09-14 21:09:01 INFO (MainThread) [hassio.store.git] Load add-on /data/addons/git/806b11b5 repository 19-09-14 21:09:01 INFO (MainThread) [hassio.store.git] Load add-on /data/addons/git/7ad98f9c repository 19-09-14 21:09:01 INFO (MainThread) [hassio.store.git] Load add-on /data/addons/git/cebe7a76 repository 19-09-14 21:09:01 INFO (MainThread) [hassio.store.git] Load add-on /data/addons/git/a722577e repository 19-09-14 21:09:01 INFO (MainThread) [hassio.store] Load add-ons from store: 68 all - 68 new - 0 remove 19-09-14 21:09:01 INFO (MainThread) [hassio.addons] Found 14 installed add-ons 19-09-14 21:09:01 INFO (SyncWorker_8) [hassio.docker.interface] Attach to hassioaddons/node-red-amd64 with version 4.1.1 19-09-14 21:09:01 INFO (SyncWorker_5) [hassio.docker.interface] Attach to hassioaddons/log-viewer-amd64 with version 0.6.3 19-09-14 21:09:01 INFO (SyncWorker_16) [hassio.docker.interface] Attach to hassioaddons/vscode-amd64 with version 0.6.5 19-09-14 21:09:01 INFO (SyncWorker_9) [hassio.docker.interface] Attach to korylprince/hassio-caddy-amd64 with version 1.6 19-09-14 21:09:01 INFO (SyncWorker_0) [hassio.docker.interface] Attach to homeassistant/amd64-addon-check_config with version 3.0 19-09-14 21:09:01 INFO (SyncWorker_17) [hassio.docker.interface] Attach to korylprince/hassio-apcupsd-amd64 with version 1.9 19-09-14 21:09:01 INFO (SyncWorker_17) [hassio.docker.interface] Attach to hassioaddons/glances-amd64 with version 0.5.5 19-09-14 21:09:01 INFO (SyncWorker_2) [hassio.docker.interface] Attach to sabeechen/hassio-google-drive-backup-amd64 with version 0.98.3 19-09-14 21:09:01 INFO (SyncWorker_3) [hassio.docker.interface] Attach to dwelch2101/zigbee2mqtt-amd64 with version 1.6.0 19-09-14 21:09:01 INFO (SyncWorker_17) [hassio.docker.interface] Attach to hassioaddons/wireguard-amd64 with version 0.1.1 19-09-14 21:09:01 INFO (SyncWorker_1) [hassio.docker.interface] Attach to homeassistant/amd64-addon-samba with version 8.1 19-09-14 21:09:01 INFO (SyncWorker_2) [hassio.docker.interface] Attach to homeassistant/amd64-addon-mosquitto with version 5.0 19-09-14 21:09:01 INFO (SyncWorker_13) [hassio.docker.interface] Attach to carldebilly/zigbee2mqttassistant with version 172 19-09-14 21:09:01 INFO (SyncWorker_6) [hassio.docker.interface] Attach to hassioaddons/sonweb-amd64 with version 0.8.2 19-09-14 21:09:01 INFO (MainThread) [hassio.updater] Fetch update data from https://version.home-assistant.io/stable.json 19-09-14 21:09:01 INFO (MainThread) [hassio.snapshots] Found 12 snapshot files 19-09-14 21:09:01 INFO (MainThread) [hassio.discovery] Load 1 messages 19-09-14 21:09:01 INFO (MainThread) [hassio.ingress] Load 0 ingress session Traceback (most recent call last): File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/src/hassio/hassio/main.py", line 47, in loop.run_until_complete(coresys.core.setup()) File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete File "/usr/src/hassio/hassio/core.py", line 76, in setup await self.sys_secrets.load() File "/usr/src/hassio/hassio/secrets.py", line 35, in load await self._read_secrets() File "/usr/src/hassio/hassio/utils/init.py", line 55, in wrapper return await method(*args, **kwargs) File "/usr/src/hassio/hassio/secrets.py", line 57, in _read_secrets _LOGGER.debug("Reload Home Assistant secrets: %s", len(self.secrets)) TypeError: object of type 'NoneType' has no len() 19-09-14 21:09:02 ERROR (MainThread) [asyncio] Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x7faba8c17950> 19-09-14 21:09:02 ERROR (MainThread) [asyncio] Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x7faba74e1b10> 19-09-14 21:09:02 ERROR (MainThread) [asyncio] Unclosed connector connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7faba71de4b0>, 1031.534)]'] connector: <aiohttp.connector.TCPConnector object at 0x7faba74e1c90>

Description of problem:

Issue started with finding I couldn't load the hassio tab in Home Assistant. I would receive this error: “Unable to load the panel source: /api/hassio/app/entrypoint.js.”

I then rebooted my host. After this, hass/homeassistant failed to return. I ssh'd into my host and found that hassio_supervisor container is cycling repeatedly. Included logs are what it outputs via 'docker logs hassio_supervisor' before it cycles over again.

I also noted that other containers may have been having issues prior to restart of host as my node-red automations had started to flake out earlier today.

I suspect this is related to 186/187 somehow; if I modify the hassio_install.sh script to set the HASSIO_VERSION=185, the container does start successfully... until it autodownloads 187 and then the issue occurs again.

There is a thread going on the forums where someone else posted what appears to be the same issue. Let me know if there is any other useful info I can provide; my Linux skills are not the strongest so I'm mainly muddling about at this point.

RaveGun commented 5 years ago

Hi, I have the same issue now with the 187.

19-09-15 01:56:16 INFO (MainThread) [__main__] Run Hass.io 19-09-15 01:56:16 INFO (MainThread) [hassio.api] Start API on 172.30.32.2 19-09-15 01:56:16 INFO (MainThread) [hassio.supervisor] Update Supervisor to version 187 19-09-15 01:56:16 INFO (SyncWorker_12) [hassio.docker.interface] Update image homeassistant/amd64-hassio-supervisor:186 to homeassistant/amd64-hassio-supervisor:187 19-09-15 01:56:16 INFO (SyncWorker_12) [hassio.docker.interface] Pull image homeassistant/amd64-hassio-supervisor tag 187. 19-09-15 01:56:24 INFO (SyncWorker_12) [hassio.docker.interface] Tag image homeassistant/amd64-hassio-supervisor with version 187 as latest 19-09-15 01:56:24 INFO (SyncWorker_12) [hassio.docker.interface] Stop hassio_supervisor application 19-09-15 01:56:24 INFO (MainThread) [__main__] Stopping Hass.io 19-09-15 01:56:24 INFO (MainThread) [hassio.api] Stop API on 172.30.32.2 19-09-15 01:56:24 INFO (MainThread) [hassio.misc.forwarder] Stop DNS forwarding 19-09-15 01:56:24 INFO (MainThread) [hassio.core] Hass.io is down 19-09-15 01:56:24 INFO (MainThread) [__main__] Close Hass.io

RaveGun commented 5 years ago

Just an addition to the issue, the 187 works in my other installation on a RPi3 + DietPi + Docker.

So it has to be related to the Ubuntu 18.04 + Docker combination.

RaveGun commented 5 years ago

It is working again.

I have deleted the 187 image and restarted the system, after I let the apt upgrade all the packages, and everything went back to normal.

ludeeus commented 5 years ago
File "/usr/src/hassio/hassio/secrets.py", line 57, in _read_secrets
_LOGGER.debug("Reload Home Assistant secrets: %s", len(self.secrets))
TypeError: object of type 'NoneType' has no len()
19-09-14 21:09:02 ERROR (MainThread) [asyncio] Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7faba8c17950>
19-09-14 21:09:02 ERROR (MainThread) [asyncio] Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7faba74e1b10>
19-09-14 21:09:02 ERROR (MainThread) [asyncio] Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7faba71de4b0>, 1031.534)]']
connector: <aiohttp.connector.TCPConnector object at 0x7faba74e1c90>

~~To get around that error create the file /config/secrets.yaml (if you do not already have it). In that file, you need at least 1 secret.~~

Example:

# In /config/secrets.yaml
# test: test

As per https://github.com/home-assistant/hassio/issues/1292#issuecomment-531565310 remove the empty secrets.yaml file.

ephirial commented 5 years ago
File "/usr/src/hassio/hassio/secrets.py", line 57, in _read_secrets
_LOGGER.debug("Reload Home Assistant secrets: %s", len(self.secrets))
TypeError: object of type 'NoneType' has no len()
19-09-14 21:09:02 ERROR (MainThread) [asyncio] Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7faba8c17950>
19-09-14 21:09:02 ERROR (MainThread) [asyncio] Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7faba74e1b10>
19-09-14 21:09:02 ERROR (MainThread) [asyncio] Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7faba71de4b0>, 1031.534)]']
connector: <aiohttp.connector.TCPConnector object at 0x7faba74e1c90>

To get around that error create the file /config/secrets.yaml (if you do not already have it). In that file, you need at least 1 secret.

Example:

# In /config/secrets.yaml
test: test

Thanks for finding this! I had the same issue, I have a generic HassIO install on Debian 10. The supervisor and core dns containers were restarting every few seconds. My secrets.yaml file was empty. After putting in "test: test", all containers started.

skawt-tee commented 5 years ago

@ludeeus Thanks! The dummy entry in secrets.yaml did the trick. I guess its fate telling me I need to get off my butt and start using the secrets.yaml.

signaleye commented 5 years ago

Have the same problem How can I get into the Path if hypervisor cannot start? keep crashing...

ludeeus commented 5 years ago

Through the host, SSH/keyboard&monitor. Jumping into the #hassio channel on discord might be the best way to get help with that :)

pvizeli commented 5 years ago

Close with 188

pvizeli commented 5 years ago

Remove the empty sectrets.yaml as workaround

matthijsberg commented 5 years ago

I have a filled secrets file (since a long time) but still crash loop of DNS and supervisor. any tips? added test: test as first line (in stead of comment) no luck. Hassio sup log shows one line only;

Post http://hassio/supervisor/update: dial tcp 172.30.32.2:80: connect: connection refused

I tried time settings, but looks good (though had a change there some time ago as my time was off in hass). After a full system reboot nothing starts anymore.

Tried this, but looks like it's for another os; https://github.com/home-assistant/hassos/blob/dev/Documentation/configuration.md

matthijsberg commented 5 years ago

Found my issue, I had a duplicate key in my secrets.yaml file. Running journalctl -fu hassio-supervisor on the docker host gave me the log where I found:

Sep 16 08:55:26 home-server hassio-supervisor[23223]: found duplicate key "nest_secret" with value "xxx" (original value: "xxx")
Sep 16 08:55:26 home-server hassio-supervisor[23223]:   in "/data/homeassistant/secrets.yaml", line 26, column 1
Sep 16 08:55:26 home-server hassio-supervisor[23223]: To suppress this check see:
Sep 16 08:55:26 home-server hassio-supervisor[23223]:     http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys
Sep 16 08:55:26 home-server hassio-supervisor[23223]: Duplicate keys will become an error in future releases, and are errors
Sep 16 08:55:26 home-server hassio-supervisor[23223]: by default when using the new API.

Removing the duplicate key solved it.

corvy commented 5 years ago

Is there any way to force a reboot remote or is a power off/on needed? My Hass.IO install is not really local to me so I need to drive some time to get it. I have SSH access and I can get to Developers tools and Configuration. Hass.IO menu is dead, portainer, configurator and Node-Red all not working (Unable to load the panel source: /api/hassio/app/entrypoint.js). Deleted my secrets.yaml from SSH but unable to cycle the OS.

core-ssh:~# hassio host reboot Post http://hassio/host/reboot: dial tcp 172.30.32.2:80: connect: connection refused

core-ssh:~# hassio homeassistant restart Processing... Done.

Post http://hassio/homeassistant/restart: dial tcp 172.30.32.2:80: connect: connection refused

Screenshot 2019-09-16 10 35 34

Seems I am left with power-cycle at this point.... unless someone knows of a backdoor.

elipinska commented 5 years ago

For those who're still facing this issue and struggle to access the host system via SSH (plus appreciate a simple step-by-step guide for dummies), here's how I managed to fix this issue on my Home Assistant:

  1. Connect your Raspberry Pi to a monitor and keyboard
  2. Access the hassio-cli by typing root, then login which will give you access to the host system
  3. Navigate (cd) to /mnt/data/supervisor/homeassistant where you should find your configuration files (huge huge thanks to @quentinsf for his uber helpful comment on this post, which saved my day here)
  4. You should then be able to use vim to open your secrets.yaml file and check for any problems (vi secrets.yaml).
  5. In my case, the file was empty, so I followed @pvizeli's guidance from this comment and just removed it (rm secrets.yaml).
  6. Finally, I exited root and ran homeassistant restart to reboot. Automations are now back, and so is my Home Assistant dashboard! 🎉
gpatkinson commented 5 years ago

Wow, I would have never found this, thanks so much for finding this bug and letting us know. I was on the verge of a full re-install (I know, not the end of the world but annoying). I had also noticed the core process in docker restarting every few seconds but am too much of a rookie with docker to know how to handle it.

DickiePhitt commented 4 years ago

Sorry to be late to the party but, from nowhere, I had the cursed “Unable to load the panel source: /api/hassio/app/entrypoint.js” when trying to access Supervisor or Files.

I've finally pulled the plug on my Rpi3 and gone for a hard reset but now HA is unable to be reached at all.

It pings OK but isn't reachable by SSH so, following the route outlined above, I've plugged a monitor and keyboard in to try and see what is happening.

The RPi boots up to the Home Assistant login prompt Sometimes, not always, I can access the host system using root login Irrespective, the RPi then closes everything down (like a reversal of boot up showing processes stopped one by one, not blank screen) then it reboots and the cycle continues. There is about 30 seconds between the login prompt appearing and the shutdown starting.

Can anyone tell me how to stop the Rpi going through this cycle?

Update: Have got through the rebooting hoop (maybe by being quick with the cd prompt) Anyway /mnt/data/supervisor/homeassistant is returning Permission Denied. Stumped so anyone tell me how to go forward?

theoneandonlyjoshua commented 3 years ago

deleting the Secrets file worked for me