Supervisor 22.03.2: Timeout while processing CodeNotary

OhioSpyderman commented 2 years ago

Describe the issue you are experiencing

What is the used version of the Supervisor?

2022.03.2

What type of installation are you running?

Home Assistant Supervised

Which operating system are you running on?

Home Assistant Operating System

What is the version of your installed operating system?

Home Assistant OS 7.4

What version of Home Assistant Core is installed?

core-2022.3.3

Steps to reproduce the issue

Restart Home Assistant 2. 3. ...

Anything in the Supervisor logs that might be useful for us?

22-03-09 10:18:31 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unknown desc = unexpected HTTP status code received from server: 500 (Internal Server Error); transport: received unexpected content-type "text/plain; charset=utf-8"

6 Times

Additional information

No response

stevenjev commented 2 years ago

Also seeing errors with CodeNotary after updating to Supervisor 2022.03.2 while running Debian 11 and Home Assistant Supervised with version core-2022.3.2:

[supervisor.utils.codenotary] Timeout while processing CodeNotary

iankaufmann commented 2 years ago

I also ran the supervisor update this morning and I'm now seeing this:

# ha core update
Error: Unknown error, see supervisor
# ha supervisor logs
22-03-09 10:44:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /info
22-03-09 10:44:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /host/info
22-03-09 10:44:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /store
22-03-09 10:44:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /core/info
22-03-09 10:44:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /supervisor/info
22-03-09 10:44:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /os/info
22-03-09 10:48:32 INFO (MainThread) [supervisor.core] Starting repair of Supervisor Environment
22-03-09 10:48:32 INFO (SyncWorker_1) [supervisor.docker] Prune stale containers
22-03-09 10:48:32 INFO (SyncWorker_1) [supervisor.docker] Prune stale images
22-03-09 10:48:36 INFO (SyncWorker_1) [supervisor.docker] Prune stale builds
22-03-09 10:48:36 INFO (SyncWorker_1) [supervisor.docker] Prune stale volumes
22-03-09 10:48:36 INFO (SyncWorker_1) [supervisor.docker] Prune stale networks
22-03-09 10:48:36 INFO (SyncWorker_1) [supervisor.docker] Fix stale container on hassio network
22-03-09 10:48:36 INFO (SyncWorker_1) [supervisor.docker] Fix stale container on host network
22-03-09 10:48:36 INFO (MainThread) [supervisor.addons] Found 1 add-ons to repair
22-03-09 10:48:36 INFO (MainThread) [supervisor.addons] Repairing for add-on: a0d7b954_aircast
22-03-09 10:48:36 INFO (SyncWorker_5) [supervisor.docker.interface] Downloading docker image ghcr.io/hassio-addons/aircast/amd64 with tag 3.2.1.
22-03-09 10:48:43 INFO (MainThread) [supervisor.supervisor] Repairing Supervisor 2022.03.2
22-03-09 10:48:43 INFO (MainThread) [supervisor.core] Finished repair of Supervisor Environment
22-03-09 10:48:57 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /available_updates
22-03-09 10:49:02 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 10:49:14 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /available_updates
22-03-09 10:49:19 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2022.3.3
22-03-09 10:49:19 INFO (SyncWorker_4) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/qemux86-64-homeassistant:2022.3.2 to ghcr.io/home-assistant/qemux86-64-homeassistant:2022.3.3
22-03-09 10:49:19 INFO (SyncWorker_4) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/qemux86-64-homeassistant with tag 2022.3.3.
22-03-09 10:49:39 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /available_updates
22-03-09 10:49:45 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 10:49:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /info
22-03-09 10:49:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /host/info
22-03-09 10:49:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /store
22-03-09 10:49:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /core/info
22-03-09 10:49:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /supervisor/info
22-03-09 10:49:55 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /os/info
22-03-09 10:50:29 ERROR (MainThread) [supervisor.jobs] Unhandled exception:
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper
    return await self._method(*args, **kwargs)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update
    await _update(version)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update
    await self.instance.update(
  File "/usr/src/supervisor/supervisor/utils/__init__.py", line 33, in wrap_api
    return await method(api, *args, **kwargs)
asyncio.exceptions.TimeoutError
22-03-09 10:50:29 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 34.207.232.93:443: i/o timeout"

mwrben commented 2 years ago

I've seen the same issue after Supervisor updated to 2022.3.2 - I now can't update core to 2022.3.3 Log:- 22-03-09 16:38:37 INFO (SyncWorker_2) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/generic-x86-64-homeassistant:2022.2.9 to ghcr.io/home-assistant/generic-x86-64-homeassistant:2022.3.3 22-03-09 16:38:37 INFO (SyncWorker_2) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/generic-x86-64-homeassistant with tag 2022.3.3. 22-03-09 16:39:33 ERROR (MainThread) [supervisor.jobs] Unhandled exception: Traceback (most recent call last): File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper return await self._method(*args, *kwargs) File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update await _update(version) File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update await self.instance.update( File "/usr/src/supervisor/supervisor/utils/init.py", line 33, in wrap_api return await method(api, args, **kwargs) asyncio.exceptions.TimeoutError 22-03-09 16:39:43 ERROR (MainThread) [supervisor.utils.codenotary] Timeout while processing CodeNotary 22-03-09 16:39:44 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2022.3.3 22-03-09 16:39:44 INFO (SyncWorker_6) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/generic-x86-64-homeassistant:2022.2.9 to ghcr.io/home-assistant/generic-x86-64-homeassistant:2022.3.3 22-03-09 16:39:44 INFO (SyncWorker_6) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/generic-x86-64-homeassistant with tag 2022.3.3.

crhass commented 2 years ago

I have the same issue on rpi 3b+ following the supervisor update

Manage supervisor gives Could not load the Supervisor panel!

Unable to restart home assistant from the settings page.

2022-03-09 18:02:33 WARNING (MainThread) [homeassistant.components.hassio] Can't read Supervisor data: 2022-03-09 18:02:33 ERROR (MainThread) [homeassistant.components.hassio.handler] /host/info return code 403 2022-03-09 18:02:33 ERROR (MainThread) [homeassistant.components.hassio.handler] /store return code 403 2022-03-09 18:02:33 ERROR (MainThread) [homeassistant.components.hassio.handler] /core/info return code 403 2022-03-09 18:02:33 ERROR (MainThread) [homeassistant.components.hassio.handler] /supervisor/info return code 403 2022-03-09 18:02:33 ERROR (MainThread) [homeassistant.components.hassio.handler] /os/info return code 403

Rep-Hueman commented 2 years ago

Logger: snitun.client.client_peer Source: /usr/local/lib/python3.9/site-packages/snitun/client/client_peer.py:104 First occurred: 18:52:22 (1 occurrences) Last logged: 18:52:22

Challenge/Response error with SniTun server (0 bytes read on a total of 32 expected bytes)

Logger: hass_nabucasa.remote Source: /usr/local/lib/python3.9/site-packages/hass_nabucasa/remote.py:333 First occurred: 18:52:22 (1 occurrences) Last logged: 18:52:22

Connection problem to snitun server

clajog commented 2 years ago

The same: Hello, I have problems with the upgrade to version 2022.3.3., currently 2022.3.1. These error messages are in the log. I tried it with 2 different systems (RPi4, 4GB and RPi4, 8GB, both with Home Assistant OS 7.4, core-2022.3.1,supervisor-2022.03.2). Restart core or Host --> same error 22-03-09 18:40:20 INFO (MainThread) [supervisor.homeassistant.module] Backup Home Assistant Core config folder done 22-03-09 18:40:20 INFO (MainThread) [supervisor.backups.manager] Creating partial backup with slug 1d3de046 completed 22-03-09 18:40:20 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2022.3.3 22-03-09 18:40:20 INFO (SyncWorker_0) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/raspberrypi4-64-homeassistant:2022.3.1 to ghcr.io/home-assistant/raspberrypi4-64-homeassistant:2022.3.3 22-03-09 18:40:20 INFO (SyncWorker_0) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/raspberrypi4-64-homeassistant with tag 2022.3.3. 22-03-09 18:42:27 ERROR (MainThread) [supervisor.jobs] Unhandled exception: Traceback (most recent call last): File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper return await self._method(args, kwargs) File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update await _update(version) File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update await self.instance.update( File "/usr/src/supervisor/supervisor/utils/init.py", line 33, in wrap_api return await method(api, args, kwargs) asyncio.exceptions.TimeoutError 22-03-09 18:42:28 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 34.207.232.93:443: i/o timeout" 22-03-09 18:42:44 INFO (MainThread) [supervisor.backups.manager] Found 8 backup files 22-03-09 18:42:44 INFO (MainThread) [supervisor.backups.manager] Found 8 backup files

adamfeldman commented 2 years ago

I'm experiencing the reported issue as well. I'm wondering if this line from the top of the logs post-restart points more directly to the root problem.

22-03-09 12:03:20 WARNING (MainThread) [supervisor.plugins.manager] Can't update plugin multicast: 
22-03-09 12:03:20 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 34.207.232.93:443: i/o timeout"
22-03-09 12:03:40 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 34.207.232.93:443: i/o timeout"

vchaindz commented 2 years ago

Due to a very heavy load, the Codenotary CAS service was overwhelmed today and was down for less than 1 hour. Resources have been substantially upgraded now. Unfortunately, we also were required to do a DNS change. Therefore, some systems that still use cached DNS entries need clear the local DNS cache. Sorry for the inconveniences, but all is back up

clajog commented 2 years ago

perfekt, it works. Thanks a lot

crhass commented 2 years ago

perfekt, it works. Thanks a lot

What did you do, I can't restart it or access SSH?

clajog commented 2 years ago

i use only ipconfig /flushdns in the commandline https://www.heise.de/tipps-tricks/DNS-Cache-loeschen-4603703.html

iankaufmann commented 2 years ago

I rolled back to a proxmox snapshot of my whole VM (HASS OS 7.4) from this morning.

Then I tried once again to update Core to 2022.3.3

I'm back in the same boat where the Supervisor is all messed up. It won't load from the UI:

Trying to open any Add-on from the sidebar says "Unable to load the panel source: /api/hassio/app/entrypoint.js."

Logs are full of "Invalid token for access".

What do I need to do to get back to a working system?

# ha supervisor logs
22-03-09 12:17:31 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.SECURITY/ContextType.CORE
22-03-09 12:17:31 INFO (MainThread) [supervisor.resolution.check] System checks complete
22-03-09 12:17:31 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
22-03-09 12:17:31 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type "text/plain; charset=utf-8"
22-03-09 12:17:31 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
22-03-09 12:17:31 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
22-03-09 12:17:31 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
22-03-09 12:18:06 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-03-09 12:18:26 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/info access from cebe7a76_hassio_google_drive_backup
22-03-09 12:18:26 INFO (MainThread) [supervisor.api.middleware.security] /backups access from cebe7a76_hassio_google_drive_backup
22-03-09 12:48:12 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-03-09 13:17:31 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.RUNNING
22-03-09 13:17:31 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.FREE_SPACE/ContextType.SYSTEM
22-03-09 13:17:31 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.PWNED/ContextType.ADDON
22-03-09 13:17:31 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.PLUGIN
22-03-09 13:17:33 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type "text/plain; charset=utf-8"
22-03-09 13:17:37 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type "text/plain; charset=utf-8"
22-03-09 13:17:39 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type "text/plain; charset=utf-8"
22-03-09 13:17:45 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type "text/plain; charset=utf-8"
22-03-09 13:17:49 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type "text/plain; charset=utf-8"
22-03-09 13:17:49 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.CORE
22-03-09 13:18:09 ERROR (MainThread) [supervisor.resolution.check] Error during processing IssueType.TRUST:
22-03-09 13:18:09 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.SUPERVISOR
22-03-09 13:18:09 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 35.168.17.115:443: i/o timeout"
22-03-09 13:18:14 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type "text/plain; charset=utf-8"
22-03-09 13:18:14 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.SECURITY/ContextType.CORE
22-03-09 13:18:14 INFO (MainThread) [supervisor.resolution.check] System checks complete
22-03-09 13:18:14 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
22-03-09 13:18:15 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type "text/plain; charset=utf-8"
22-03-09 13:18:16 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
22-03-09 13:18:16 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
22-03-09 13:18:16 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
22-03-09 13:18:18 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-03-09 13:18:27 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/info access from cebe7a76_hassio_google_drive_backup
22-03-09 13:18:27 INFO (MainThread) [supervisor.api.middleware.security] /backups access from cebe7a76_hassio_google_drive_backup
22-03-09 13:18:29 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
22-03-09 13:18:33 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type "text/plain; charset=utf-8"
22-03-09 13:25:09 INFO (MainThread) [supervisor.host.info] Updating local host information
22-03-09 13:25:10 INFO (MainThread) [supervisor.host.services] Updating service information
22-03-09 13:25:10 INFO (MainThread) [supervisor.host.network] Updating local network information
22-03-09 13:25:10 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
22-03-09 13:25:10 INFO (MainThread) [supervisor.host.manager] Host information reload completed
22-03-09 13:48:24 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-03-09 14:07:41 INFO (SyncWorker_4) [supervisor.docker] Runing command 'python3 -m homeassistant -c /config --script check_config' on ghcr.io/home-assistant/qemux86-64-homeassistant
22-03-09 14:08:49 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2022.3.3
22-03-09 14:08:49 ERROR (MainThread) [supervisor.utils] Can't execute update while a task is in progress
22-03-09 14:08:51 INFO (MainThread) [supervisor.homeassistant.module] Update pulse/client.config: /data/tmp/homeassistant_pulse
22-03-09 14:08:51 ERROR (MainThread) [supervisor.utils] Can't execute run while a task is in progress
22-03-09 14:08:51 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready
22-03-09 14:08:56 INFO (MainThread) [supervisor.homeassistant.core] Detect a running Home Assistant instance
22-03-09 14:08:56 INFO (MainThread) [supervisor.homeassistant.core] Successfully started Home Assistant 2022.3.3
22-03-09 14:08:56 ERROR (MainThread) [supervisor.utils] Can't execute cleanup while a task is in progress
22-03-09 14:09:07 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /available_updates
22-03-09 14:09:09 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:09:16 INFO (MainThread) [supervisor.homeassistant.core] Home Assistant config is valid
22-03-09 14:09:19 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:09:27 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:09:42 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:09:47 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /available_updates
22-03-09 14:09:56 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:09:58 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:12:17 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:12:20 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /available_updates
22-03-09 14:12:27 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:12:29 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /available_updates
22-03-09 14:13:48 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /available_updates
22-03-09 14:13:50 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:13:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /info
22-03-09 14:13:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /host/info
22-03-09 14:13:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /store
22-03-09 14:13:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /core/info
22-03-09 14:13:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /supervisor/info
22-03-09 14:13:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /os/info
22-03-09 14:13:52 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /available_updates
22-03-09 14:14:11 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /app/entrypoint.js
22-03-09 14:15:32 INFO (MainThread) [supervisor.plugins.dns] Restarting CoreDNS plugin
22-03-09 14:15:32 INFO (SyncWorker_1) [supervisor.docker.interface] Restarting ghcr.io/home-assistant/amd64-hassio-dns

BebeMischa commented 2 years ago

on Debian Bullseye with Supervised HA this saved my day:

sudo systemd-resolve --flush-caches
sudo /etc/init.d/networking restart

davidrustingha commented 2 years ago

@BebeMischa did that fix not being able to update for you? For me it threw a diffrent error

22-03-09 21:35:26 ERROR (MainThread) [supervisor.jobs] Unhandled exception: 
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper
    return await self._method(*args, **kwargs)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update
    await _update(version)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update
    await self.instance.update(
  File "/usr/src/supervisor/supervisor/utils/__init__.py", line 33, in wrap_api
    return await method(api, *args, **kwargs)
asyncio.exceptions.TimeoutError
22-03-09 21:35:36 ERROR (MainThread) [supervisor.utils.codenotary] Timeout while processing CodeNotary

gentigit commented 2 years ago

On hassio it started working after:

ha dns update
ha dns restart

might have been a coincidence too, and the upstream DNS happened to update around the same time.

BebeMischa commented 2 years ago

@BebeMischa did that fix not being able to update for you? For me it threw a diffrent error

I was able to update, but after that my system was marked unsupported. That issue is now gone.

iankaufmann commented 2 years ago

After ha dns restart (it was already up to date) I am getting the same as @davidrustingha

22-03-09 14:50:38 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2022.3.3
22-03-09 14:50:38 INFO (SyncWorker_0) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/qemux86-64-homeassistant:2022.3.2 to ghcr.io/home-assistant/qemux86-64-homeassistant:2022.3.3
22-03-09 14:50:38 INFO (SyncWorker_0) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/qemux86-64-homeassistant with tag 2022.3.3.
22-03-09 14:50:58 ERROR (MainThread) [supervisor.jobs] Unhandled exception:
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper
    return await self._method(*args, **kwargs)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update
    await _update(version)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update
    await self.instance.update(
  File "/usr/src/supervisor/supervisor/utils/__init__.py", line 33, in wrap_api
    return await method(api, *args, **kwargs)
asyncio.exceptions.TimeoutError
22-03-09 14:50:58 ERROR (MainThread) [supervisor.utils.codenotary] Timeout while processing CodeNotary

Does anybody know what DNS issue is causing this?

From my firewall, cas.codenotary.com pings to 35.168.17.115, which appears correct according to MXToolbox, but I don't really know for sure what I'm looking for here.

That was the only domain that I saw my home assistant box looking for in my DNS logs.

jackyaz commented 2 years ago

After ha dns restart (it was already up to date) I am getting the same as @davidrustingha

22-03-09 14:50:38 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2022.3.3
22-03-09 14:50:38 INFO (SyncWorker_0) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/qemux86-64-homeassistant:2022.3.2 to ghcr.io/home-assistant/qemux86-64-homeassistant:2022.3.3
22-03-09 14:50:38 INFO (SyncWorker_0) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/qemux86-64-homeassistant with tag 2022.3.3.
22-03-09 14:50:58 ERROR (MainThread) [supervisor.jobs] Unhandled exception:
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper
    return await self._method(*args, **kwargs)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update
    await _update(version)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update
    await self.instance.update(
  File "/usr/src/supervisor/supervisor/utils/__init__.py", line 33, in wrap_api
    return await method(api, *args, **kwargs)
asyncio.exceptions.TimeoutError
22-03-09 14:50:58 ERROR (MainThread) [supervisor.utils.codenotary] Timeout while processing CodeNotary

Does anybody know what DNS issue is causing this?

From my firewall, cas.codenotary.com pings to 35.168.17.115, which appears correct according to MXToolbox, but I don't really know for sure what I'm looking for here.

That was the only domain that I saw my home assistant box looking for in my DNS logs.

I'm seeing this too, I opened a separate issue as I'm not sure if its the same as this one or not https://github.com/home-assistant/supervisor/issues/3496

TomBeech85 commented 2 years ago

I was experiencing the OPs issue. I updated my DNS and now am getting this error:

22-03-09 21:15:46 INFO (SyncWorker_9) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/qemux86-64-homeassistant:2022.2.9 to ghcr.io/home-assistant/qemux86-64-homeassistant:2022.3.3 22-03-09 21:15:46 INFO (SyncWorker_9) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/qemux86-64-homeassistant with tag 2022.3.3. 22-03-09 21:15:47 ERROR (MainThread) [supervisor.resolution.check] Error during processing IssueType.TRUST: 22-03-09 21:15:47 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.SECURITY/ContextType.CORE 22-03-09 21:15:47 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.CORE 22-03-09 21:15:47 ERROR (MainThread) [supervisor.utils] Can't execute check_trust while a task is in progress 22-03-09 21:15:47 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.PWNED/ContextType.ADDON 22-03-09 21:15:47 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.FREE_SPACE/ContextType.SYSTEM 22-03-09 21:15:47 INFO (MainThread) [supervisor.resolution.module] Create new suggestion SuggestionType.CREATE_FULL_BACKUP - ContextType.SYSTEM / None 22-03-09 21:15:47 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.SUPERVISOR 22-03-09 21:15:49 ERROR (MainThread) [supervisor.utils.codenotary] Can't parse CodeNotary output: b'' - Expecting value: line 1 column 1 (char 0) 22-03-09 21:15:53 ERROR (MainThread) [supervisor.utils.codenotary] Can't parse CodeNotary output: b'' - Expecting value: line 1 column 1 (char 0)

adrianmihalko commented 2 years ago

I am trying to update from 2022.2.6 to 2022.3.3, but I am getting error:

22-03-09 22:18:43 ERROR (MainThread) [supervisor.jobs] Unhandled exception: 
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper
    return await self._method(*args, **kwargs)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update
    await _update(version)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update
    await self.instance.update(
  File "/usr/src/supervisor/supervisor/utils/__init__.py", line 33, in wrap_api
    return await method(api, *args, **kwargs)
asyncio.exceptions.TimeoutError
22-03-09 22:18:43 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 35.168.17.115:443: i/o timeout"
22-03-09 22:18:45 ERROR (MainThread) [supervisor.utils.codenotary] Timeout while processing CodeNotary
22-03-09 22:19:06 ERROR (MainThread) [supervisor.utils.codenotary] Timeout while processing CodeNotary
22-03-09 22:19:06 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
22-03-09 22:19:06 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
22-03-09 22:19:06 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete

Supervisor is already the latest: supervisor-2022.03.2

sermayoral commented 2 years ago

Is there any way to downgrade the supervisor in a supervised installation?

gentigit commented 2 years ago

Hmm, I was getting the same IP before so maybe codenotary is still getting hammered and I happened to get through. Seeing this 35.168.17.115:443: i/o timeout Also, ha loves to use 1.1.1.1:853 for DNS.

nodomain commented 2 years ago

Can you please reopen https://github.com/codenotary/cas/issues/263 @frenck - I now also get the rpc errors again due to timeout. Thx

nodomain commented 2 years ago

Got through now. Perhaps related to ´us-east-1´ AWS outage?

BebeMischa commented 2 years ago

Got through now. Perhaps related to ´us-east-1´ AWS outage?

35.168.17.115 is registered by Amazon... so you could be right ;-)

jmound commented 2 years ago

22-03-09 15:42:04 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 35.168.17.115:443: i/o timeout" That's CST

BebeMischa commented 2 years ago

jurekmff commented 2 years ago

For me its still the same issue, trying to update from core-2022.2.9 to core-2022.3.3 with supervisor-2022.03.2 and Home Assistant OS 7.4.

22-03-09 23:29:02 INFO (SyncWorker_2) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/raspberrypi4-64-homeassistant with tag 2022.3.3.
22-03-09 23:29:23 ERROR (MainThread) [supervisor.jobs] Unhandled exception: 
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper
    return await self._method(*args, **kwargs)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update
    await _update(version)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update
    await self.instance.update(
  File "/usr/src/supervisor/supervisor/utils/__init__.py", line 33, in wrap_api
    return await method(api, *args, **kwargs)
asyncio.exceptions.TimeoutError
22-03-09 23:29:33 ERROR (MainThread) [supervisor.utils.codenotary] Timeout while processing CodeNotary

msolomos commented 2 years ago

For me also the same issue, trying to update from core-2022.2.9 to core-2022.3.3 with supervisor-2022.03.2 and Home Assistant OS 7.4.

22-03-10 00:30:28 ERROR (MainThread) [supervisor.jobs] Unhandled exception: Traceback (most recent call last): File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper return await self._method(*args, **kwargs) File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update await _update(version) File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update await self.instance.update( File "/usr/src/supervisor/supervisor/utils/__init__.py", line 33, in wrap_api return await method(api, *args, **kwargs) asyncio.exceptions.TimeoutError 22

dnedeltchev commented 2 years ago

Same here, and I REALLY REALLY REALLY REALLY REALLY REALLY REALLY REALLY wish that you stop this supervisor auto update! Come on! I usually have a DNS block to stop this but had left it unblocked and got caught out with another supervisor auto update! PLEASE STOP THIS AUTO UPDATE! This is really getting silly!

OhioSpyderman commented 2 years ago

On paper (from the release notes), this seemed pretty harmless. Why is this snowballing?

Bob.

jmound commented 2 years ago

The change introduced a new, well-intentioned, likely valuable, cloud based dependency. The ability to disable it or have some method of working around that dependency would be helpful.

OhioSpyderman commented 2 years ago

Work for Nabu Casa?

boomam commented 2 years ago

Same issue for me too, DNS cache flushed.

22-03-09 18:33:37 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 35.168.17.115:443: i/o timeout"
22-03-09 18:45:24 ERROR (MainThread) [supervisor.jobs] Unhandled exception: 
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper
    return await self._method(*args, **kwargs)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update
    await _update(version)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update
    await self.instance.update(
  File "/usr/src/supervisor/supervisor/utils/__init__.py", line 33, in wrap_api
    return await method(api, *args, **kwargs)
asyncio.exceptions.TimeoutError
22-03-09 18:45:34 ERROR (MainThread) [supervisor.utils.codenotary] Timeout while processing CodeNotary

tjhorner commented 2 years ago

Having issues updating to the latest core:

22-03-09 18:41:40 ERROR (MainThread) [supervisor.jobs] Unhandled exception: 
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper
    return await self._method(*args, **kwargs)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 229, in update
    await _update(version)
  File "/usr/src/supervisor/supervisor/homeassistant/core.py", line 207, in _update
    await self.instance.update(
  File "/usr/src/supervisor/supervisor/utils/__init__.py", line 33, in wrap_api
    return await method(api, *args, **kwargs)
asyncio.exceptions.TimeoutError
22-03-09 18:41:40 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 35.168.17.115:443: i/o timeout"

Update: after clearing DNS cache at both the machine and network level, the last line changed to this:

22-03-09 18:50:36 ERROR (MainThread) [supervisor.utils.codenotary] Timeout while processing CodeNotary

jbishop129 commented 2 years ago

Unable to complete a Supervised install on an ARM SoC. I suppose it'll work itself out eventually when the underlying issue is resolved

22-03-09 16:46:44 INFO (SyncWorker_4) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/qemuarm-64-homeassistant with tag 2022.3.3.
22-03-09 16:47:05 WARNING (MainThread) [supervisor.homeassistant.core] Error on Home Assistant installation. Retry in 30sec
22-03-09 16:47:05 WARNING (MainThread) [supervisor.utils.codenotary] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 35.168.17.115:443: i/o timeout"

iankaufmann commented 2 years ago

Will getting Core to update to 2022.3.3 also fix what is going on with the supervisor panel and addons? I don't care about putting off the update to 2022.3.3 until whatever this is is fixed, but at the moment, I can't load the supervisor panel at all, or any addon panel (esphome, z-wavejs, etc) and that is what makes this a more stressful issue.

22-03-09 17:38:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /info
22-03-09 17:38:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /host/info
22-03-09 17:38:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /store
22-03-09 17:38:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /core/info
22-03-09 17:38:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /supervisor/info
22-03-09 17:38:51 ERROR (MainThread) [supervisor.api.middleware.security] Invalid token for access /os/info

a-bianucci commented 2 years ago

My HASS crashed today for some reason..... I ended up wiping it and was going to reinstall from last nights backup which failed several times.

Tried a complete reinstall and I'm stuck with the Timeout while processing CodeNotary and it's at the loading, must wait up to 20m screen.

Supervisor logs just so a restart of the homeassistant container every few minutes but no progress

Dying here, what on earth is going on today.

Mpgod80 commented 2 years ago

Same here, and I REALLY REALLY REALLY REALLY REALLY REALLY REALLY REALLY wish that you stop this supervisor auto update! Come on! I usually have a DNS block to stop this but had left it unblocked and got caught out with another supervisor auto update! PLEASE STOP THIS AUTO UPDATE! This is really getting silly!

I could not agree with you more . Stop the Auto update of Supervisor!! Not the first time it breaks HA...

jpawns commented 2 years ago

I could reload a snapshot from last night. Is there a way to prevent supervisor from updating?

iankaufmann commented 2 years ago

I could reload a snapshot from last night. Is there a way to prevent supervisor from updating?

I restored from a VM snapshot on proxmox and supervisor updated to latest shortly after

lianmengwa commented 2 years ago

VM snapshot is not working for me.

jasonrwise77 commented 2 years ago

Same issue for me as well, is there a fix?

RomRider commented 2 years ago

There's nothing you can do at the moment. CodeNotary seems to be super slow to respond. It's not on HA's side. The codenotary service needs to be restored properly before it will work again and it's not HA's team owning it.

RomRider commented 2 years ago

Super ugly hack that I DON'T RECOMMEND AT ALL to anyone because you completely bypass the image check! ONLY DO THIS IF YOU CAN'T WAIT AT ALL!!

Everything is done as root.

# On the machine running HA
docker exec -it hassio_supervisor sh
# in the container
vi /usr/src/supervisor/supervisor/utils/codenotary.py

You'll want the beginning of the cas_validate function to look like this (in the docker container):

async def cas_validate(
    signer: str,
    checksum: str,
) -> None:
    """Validate data against CodeNotary."""
    if (checksum, signer) in _CACHE:
        return

    _CACHE.add((checksum, signer))  # This line was added
    return                          # This line was added

Then

# exit the container and
service hassio-supervisor restart

mr-varga commented 2 years ago

For me solved: ha core rebuild

chaosye0 commented 2 years ago

fine, after the upgrade I have this error too. maybe this "Home Assistant Operating System" does not suit me, any other trouble-less options to recommend, my friends? maybe "Home Assistant Core" or "Home Assistant Container"?

wrkn4alivn commented 2 years ago

EDIT: Confirmed that these symptoms self-resolved after several hours & updating supervisor to 22.03.3.

Could this issue explain why HA can no longer reach Unifi or ZWaveJS2MQTT addons? Seems to coincide with the autoupdate of Supervisor, and restarting the addons & reloading integrations doesn't seem to make a difference.

Failed to connect: Cannot connect to host a0d7b954-zwavejs2mqtt:3000 ssl:default [Connect call failed ('172.30.33.3', 3000)]

Can't read Supervisor data: 404 Client Error for http+docker://localhost/v1.41/containers/addon_a0d7b954_unifi/json: Not Found ("No such container: addon_a0d7b954_unifi")

HaTaX2 commented 2 years ago

On Home Assistant OS 7.4 here and ran into this updating from 2022.3.2 to 2022.3.3. At first it started with my system being unhealthy and I couldn't apply the update so I restarted the entire VM, then I noticed all supervisor related functions were no longer working and the supervisor panel wouldn't load anymore.

In my supervisor logs I found a lot of '[supervisor.api.middleware.security] Invalid token for access' messages and that led me to do a 'ha core rebuild' and 'ha host reboot'. After that all my supervisor stuff looks to be working and logs are healthy again, but it runs into the same issue everyone else reports when trying to update to 2022.3.3. '[supervisor.utils.codenotary] Timeout while processing CodeNotary' after the stacktrace for 'asyncio.exceptions.TimeoutError'.

Sounds like a server side error that's out of everyone's hands at the moment, I'll just watch this thread for updates.

LostSoulfly commented 2 years ago

This is causing so many issues for me. For a "local-first" solution to home automation, this brief outage shouldn't be causing so much mayhem.

home-assistant / supervisor