Websocket Errors On Supervisor 2023.04.1 and OS 10.0 . Unable to connect to Home Assistant

[townsmcp]

Describe the issue you are experiencing

Since updating to Supervisor 2023.03.2 yesterday, from various devices running mobile Safari and the HA Companion app I keep getting the following websocket errors:

Error: Custom response validation failed with error: 500, 500 Internal Server Error Server got itself in trouble Retry Count: 5 Next Retry: 3:28:41 pm

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

1. Try to log into HA on mobile safari
2. Try to log into HA from Companion app (latest App Store version or Test Flight version 2023.3 (2023.455)
3. Click Retry. Observe same error message
4. Try to log in from different devices as different users and observe same behaviour

Anything in the Supervisor logs that might be useful for us?

Nothing obvious to me however this is the log from the last few minutes

23-03-22 15:56:40 WARNING (MainThread) [supervisor.addons.options] Option 'blocklist' does not exist in the schema for Zigbee2MQTT (45df7312_zigbee2mqtt)
23-03-22 15:56:40 WARNING (MainThread) [supervisor.addons.options] Option 'passlist' does not exist in the schema for Zigbee2MQTT (45df7312_zigbee2mqtt)
23-03-22 15:56:40 WARNING (MainThread) [supervisor.addons.options] Option 'queue' does not exist in the schema for Zigbee2MQTT (45df7312_zigbee2mqtt)
23-03-22 15:56:40 WARNING (MainThread) [supervisor.addons.options] Option 'frontend' does not exist in the schema for Zigbee2MQTT (45df7312_zigbee2mqtt)
23-03-22 15:56:40 WARNING (MainThread) [supervisor.addons.options] Option 'experimental' does not exist in the schema for Zigbee2MQTT (45df7312_zigbee2mqtt)
23-03-22 15:56:40 WARNING (MainThread) [supervisor.addons.options] Option 'availability' does not exist in the schema for Zigbee2MQTT (45df7312_zigbee2mqtt)

### System Health information

## System Information

version | core-2023.3.5
-- | --
installation_type | Home Assistant OS
dev | false
hassio | true
docker | true
user | root
virtualenv | false
python_version | 3.10.10
os_name | Linux
os_version | 5.15.90
arch | x86_64
timezone | Europe/London
config_dir | /config

<details><summary>Home Assistant Community Store</summary>

GitHub API | ok
-- | --
GitHub Content | ok
GitHub Web | ok
GitHub API Calls Remaining | 5000
Installed Version | 1.31.0
Stage | running
Available Repositories | 1261
Downloaded Repositories | 91
HACS Data | ok

</details>

<details><summary>Home Assistant Cloud</summary>

logged_in | true
-- | --
subscription_expiration | 28 February 2024 at 12:00 am
relayer_connected | true
remote_enabled | true
remote_connected | true
alexa_enabled | true
google_enabled | true
remote_server | eu-central-1-6.ui.nabu.casa
can_reach_cert_server | ok
can_reach_cloud_auth | ok
can_reach_cloud | ok

</details>

<details><summary>Home Assistant Supervisor</summary>

host_os | Home Assistant OS 9.5
-- | --
update_channel | beta
supervisor_version | supervisor-2023.03.2
agent_version | 1.4.1
docker_version | 20.10.22
disk_total | 97.7 GB
disk_used | 21.0 GB
healthy | true
supported | true
board | ova
supervisor_api | ok
version_api | ok
installed_addons | File editor (5.5.0), AirCast (3.5.2), Samba share (10.0.0), AirSonos (3.5.2), Log Viewer (0.14.0), Node-RED (14.1.0), phpMyAdmin (0.8.4), Studio Code Server (5.5.4), Terminal & SSH (9.6.1), Home Assistant Google Drive Backup (0.110.1), Portainer (2.0.0), Mosquitto broker (6.1.3), MariaDB (2.5.2), Tailscale (0.7.0), Cloudflared (4.0.13), Zigbee2MQTT (1.30.2-1), thesillyhome addon dev (2.0.13-dev), Zigbee2MQTT Edge (edge), Uptime Kuma (0.7.0), Matter Server (4.1.0)

</details>

<details><summary>Dashboards</summary>

dashboards | 4
-- | --
resources | 56
views | 58
mode | storage

</details>

<details><summary>Recorder</summary>

oldest_recorder_run | 21 March 2023 at 5:30 pm
-- | --
current_recorder_run | 22 March 2023 at 1:49 pm
estimated_db_size | 759.80 MiB
database_engine | mysql
database_version | 10.6.10

</details>

<details><summary>Solcast PV Forecast</summary>

can_reach_server | ok
-- | --
remaining_requests | 13
rooftop_site_count | 1
polling_hours | 6 -> 19 and once at midnight

</details>

### Supervisor diagnostics

_No response_

### Additional information

_No response_

[townsmcp]

Supervisor diagnostic file: config_entry-hassio-b5837ff0945f39a3720ae608ccdfa0c5.json.txt

[townsmcp]

I have just found this in Core logs:

Logger: aiohttp.server
Source: auth/__init__.py:536
First occurred: 4:11:32 pm (8 occurrences)
Last logged: 4:11:33 pm

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 67, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 80, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 235, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 146, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 95, in handle_req
    resp = await func(view, request, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/auth/__init__.py", line 258, in post
    return await self._async_handle_refresh_token(hass, data, request.remote)
  File "/usr/src/homeassistant/homeassistant/components/auth/__init__.py", line 375, in _async_handle_refresh_token
    access_token = hass.auth.async_create_access_token(
  File "/usr/src/homeassistant/homeassistant/auth/__init__.py", line 504, in async_create_access_token
    self.async_validate_refresh_token(refresh_token, remote_ip)
  File "/usr/src/homeassistant/homeassistant/auth/__init__.py", line 550, in async_validate_refresh_token
    if provider := self._async_resolve_provider(refresh_token):
  File "/usr/src/homeassistant/homeassistant/auth/__init__.py", line 536, in _async_resolve_provider
    raise InvalidProvider(
homeassistant.auth.InvalidProvider: Auth provider homeassistant, None not available

And:

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:53
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 4:11:37 pm (8 occurrences)
Last logged: 4:13:09 pm

[140628374256896] Disconnected: Did not receive auth message within 10 seconds
[140628315628800] Disconnected: Did not receive auth message within 10 seconds
[140628328886624] Disconnected: Did not receive auth message within 10 seconds
[140628417039472] Disconnected: Did not receive auth message within 10 seconds
[140628315624480] Disconnected: Did not receive auth message within 10 seconds

[Mariusthvdb]

can I add this:

Logger: homeassistant.components.hassio
Source: components/hassio/websocket_api.py:125 
Integration: Home Assistant Supervisor (documentation, issues) 
First occurred: 00:19:13 (3 occurrences) 
Last logged: 00:19:13

Failed to to call /ingress/validate_session -

and

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:470 
Integration: Home Assistant Supervisor (documentation, issues) 
First occurred: 00:19:13 (3 occurrences) 
Last logged: 00:19:13

/ingress/validate_session return code 401

started after the same update to 2023.03.2

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[townsmcp]

Still an issue. My install versions are:

Home Assistant 2023.4.6 Supervisor 2023.04.1 Operating System 10.0 Frontend 20230411.1 - latest

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.