Time out on stats request

wtstreetglow commented 2 years ago

The problem

I am getting the below error in my logs.

What version of Home Assistant Core has the issue?

2022.6.7

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Hassio

Link to integration documentation on our website

https://www.home-assistant.io/integrations/hassio

Diagnostics information

`Logger: homeassistant.components.hassio.handler Source: components/hassio/handler.py:263 Integration: Home Assistant Supervisor (documentation, issues) First occurred: 9:21:24 AM (9 occurrences) Last logged: 9:21:24 AM

Timeout on /addons/core_zwave_js/stats request Timeout on /addons/core_duckdns/stats request Timeout on /addons/a0d7b954_vscode/stats request Timeout on /addons/core_configurator/stats request Timeout on /addons/a0d7b954_ssh/stats request`

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

Hey there @home-assistant/supervisor, mind taking a look at this issue as it has been labeled with an integration (hassio) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

hassio documentation hassio source _{^{(message by IssueLinks)}}

thesilentmiaow commented 2 years ago

I'm seeing this too

cortado commented 2 years ago

Same here, lots of Supervisor errors there days.

Home Assistant 2022.8.6 Supervisor 2022.08.3 Operating System 8.4 Grensesnitt 20220802.0 – latest Hardware: RPi3b+ (32 bit)

Logger: homeassistant.components.hassio.handler Source: components/hassio/handler.py:42 Integration: Home Assistant Supervisor (documentation, issues) First occurred: 08:33:01 (4 occurrences) Last logged: 12:28:30

Timeout on /addons/db21ed7f_filebrowser/stats request Timeout on /addons/core_zwave_js/stats request Timeout on /addons/core_mariadb/stats request /store return code 500

kubbur commented 2 years ago

Same on ha os 22.8.5

kubbur commented 2 years ago

Logger: homeassistant.components.hassio.handler Source: components/hassio/handler.py:42 Integration: Home Assistant Supervisor (documentation, issues) First occurred: 21 August 2022, 11:29:09 (20 occurrences) Last logged: 10:01:06

Timeout on /addons/a0d7b954_adguard/stats request Timeout on /addons/45df7312_zigbee2mqtt/stats request Timeout on /addons/core_ssh/stats request Timeout on /addons/core_mosquitto/stats request

tlanglois1111 commented 2 years ago

Same issue. When this happens I can no longer load the Supervisor logs in the UI and the Add Ons page may become unresponsive. If I try to restart HA, there is a timeout error from the Supervisor. I feel like the Supervisor, or something it is communicating with, is getting into some odd state where it becomes unresponsive.

Could not fetch stats for core_zwave_js: Could not fetch stats for core_configurator: Could not fetch stats for core_ssh: Could not fetch stats for a0d7b954_nodered: Could not fetch stats for 424ccef4_crowdsec-firewall-bouncer

Logger: homeassistant.components.hassio.handler Source: components/hassio/handler.py:29 Integration: Home Assistant Supervisor (documentation, issues) First occurred: 11:05:34 AM (1 occurrences) Last logged: 11:05:34 AM

Timeout on /homeassistant/restart request

Home Assistant 2022.8.5 Supervisor 2022.08.3 Operating System 8.5 Frontend 20220802.0 - latest

kedante commented 2 years ago

Same

Logger: homeassistant.components.hassio.handler Source: components/hassio/handler.py:42 Integration: Home Assistant Supervisor (documentation, [issues] First occurred: 04:20:04 (9 occurrences) Last logged: 04:20:04

Timeout on /addons/a0d7b954_logviewer/stats request Timeout on /addons/core_samba/stats request Timeout on /addons/core_mosquitto/stats request Timeout on /addons/15d21743_samba_backup/stats request Timeout on /addons/a0d7b954_nodered/stats request

Home Assistant 2022.8.7 Supervisor 2022.08.5 Operating System 8.5 Frontend 20220802.0 - latest

Respberry Pi 3 (32-bit)

besiktas97 commented 2 years ago

Also getting the same when I am trying to see the logs for decons add-on:

Logger: homeassistant.components.hassio.handler Source: components/hassio/handler.py:42 Integration: Home Assistant Supervisor (documentation, issues) First occurred: 17:17:40 (7 occurrences) Last logged: 17:17:42

Timeout on /addons/core_configurator/stats request Timeout on /addons/15d21743_samba_backup/stats request Timeout on /addons/5c53de3b_esphome-dev/stats request Timeout on /addons/5c53de3b_esphome/stats request Timeout on /addons/03b5fe11_ssh/stats request

it takes ages before I can see the logs as end result above error.

tlanglois1111 commented 2 years ago

I'm running HA on a RPi3b+ (32 bit). So after adding sensors for cpu/memory/disk for tracking performance, I noticed a few things. The biggest issue was during heavy loads (such as when the recorder runs a purge), the memory/swap is pegged close to 100%. The extra load of the mem swap plus whatever else is going on, causes a lot of thrashing and slow downs.

I then looked at where the memory was going. I switched to MariaDB a while back for performance reasons and I suspect it performs better because it caches data (using memory). I also added Node-Red to try more complicated automations. These two addons together use about 20+% of memory and I believe the extra use of mem swap due to low memory negates any performance gains from mariadb. So I switched back to sqllite and moved my node-red automations to pyscript. My memory usage went way down and the system runs a lot better.

So the moral of my story is, if you are seeing these stat request timeouts, take a look at your memory and cpu utilization over time. This may not be the root of the problem people are seeing in this thread, but it may be related. In my case, my poor old Pi 3 wasn't up to the task.

codemannh commented 1 year ago

I just updated from 2022.9.1 -> 2022.9.2 and now getting the same issue. Not sure if it is related, but all my devices that connect through mqtt are very slow to respond to a state change as well as slowness throughout HA, frontend, displaying the logs in the interface, etc. Here is the entry from todays log.

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:42
Integration: Home Assistant Supervisor
First occurred: 9:23:22 AM (30 occurrences)
Last logged: 10:03:30 AM

Timeout on /addons/core_mariadb/stats request
Timeout on /addons/a0d7b954_ssh/stats request
Timeout on /addons/67f4b1af_weatherflow2mqtt/stats request
Timeout on /addons/5c53de3b_esphome/stats request
Timeout on /addons/a0d7b954_adguard/stats request

I run HAOS in a VM on a Proxmox server. The new version of HA shows memory and CPU utilization, but they don't match up with what Proxmox is providing, but both of them show that neither memory nor CPU are an issue.

Dutchy-79 commented 1 year ago

Same problem

johny-mnemonic commented 1 year ago

Same issue here:

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:42
Integration: Home Assistant Supervisor
First occurred: 14:08:37 (11 occurrences)
Last logged: 14:08:39

Timeout on /addons/core_configurator/stats request
Timeout on /addons/a0d7b954_nginxproxymanager/stats request
Timeout on /addons/15d21743_samba_backup/stats request
Timeout on /addons/a0d7b954_esphome/stats request
Timeout on /addons/core_mariadb/stats request

I have one observation not mentioned here thought. I am also on RPi3B with 1GB of memory, but I haven't seen these issues at all till I was still running my HA on old 32bit Raspbian based on Debian 9. Sure, the Supervisor was complaining about running on unsupported OS, but everything worked. Then with 2022.9 it refused to update any addon due to OS using too old Docker, so I finally found time to wipe the SSD and install Debian 11 64bit. Since then I am having a lot of timeout issues not only on stats resources from Supervisor but from other addons as well. I am not sure whether it is caused solely by Debian 11 or the fact I went 64bit, but seems like newer OS means more strain on our old RPi3 which struggles to handle it🤷‍♂️

nao-pon commented 1 year ago

Same issue here:

Home Assistant 2022.11.2
Supervisor 2022.10.2
Operating System 9.3
Frontend 20221108.0 - latest
Hardware: rpi4-64

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:271
Integration: Home Assistant Supervisor ([documentation](https://www.home-assistant.io/integrations/hassio), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+hassio%22))
First occurred: 10:56:39 (2 occurrences)
Last logged: 10:56:40

Timeout on /core/stats request
Timeout on /supervisor/stats request

Logger: homeassistant.components.hassio
Source: components/hassio/websocket_api.py:124
Integration: Home Assistant Supervisor ([documentation](https://www.home-assistant.io/integrations/hassio), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+hassio%22))
First occurred: 10:56:40 (2 occurrences)
Last logged: 10:56:40

Failed to to call /core/stats -
Failed to to call /supervisor/stats -

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:42
Integration: Home Assistant Supervisor ([documentation](https://www.home-assistant.io/integrations/hassio), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+hassio%22))
First occurred: 10:58:11 (8 occurrences)
Last logged: 10:58:11

Timeout on /addons/a0d7b954_ssh/stats request
Timeout on /addons/cebe7a76_hassio_google_drive_backup/stats request
Timeout on /addons/4b7969ab_mosquitto/stats request
Timeout on /addons/a0d7b954_wireguard/stats request
Timeout on /addons/core_dnsmasq/stats request

docsapro commented 1 year ago

same for me (VMware VM) happens randomly also I loose the weather from met.no

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:42
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 03:29:50 (1438 occurrences)
Last logged: 18:30:36

Timeout on /host/info request
Timeout on /info request
Timeout on /supervisor/info request
Timeout on /core/info request
Timeout on /os/info request

Screenshot 2022-11-15 at 18 37 51

nothing wrong in the supervisor container restarting container doesnt help either to recover


| |  | |                          /\           (_)   | |            | |  
| |__| | ___  _ __ ___   ___     /  \   ___ ___ _ ___| |_ __ _ _ __ | |_ 
|  __  |/ _ \| '_ \ _ \ / _ \   / /\ \ / __/ __| / __| __/ _\ | '_ \| __|
| |  | | (_) | | | | | |  __/  / ____ \\__ \__ \ \__ \ || (_| | | | | |_ 
|_|  |_|\___/|_| |_| |_|\___| /_/    \_\___/___/_|___/\__\__,_|_| |_|\__|

System information
  IPv4 addresses for enp3s0: 10.0.1.41/24
  IPv6 addresses for enp3s0: xxx:1215:9c5c:1357:721e/64, xxxx:bebe:bebe:bebe:bee9/128, fe80::88a5:5f9e:63cf:6cac/64

  OS Version:               Home Assistant OS 9.3
  Home Assistant Core:      2022.11.2

➜  ~ docker logs -f --tail 550 hassio_supervisor

22-11-15 02:40:04 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'mqtt2'
22-11-15 02:40:05 INFO (MainThread) [supervisor.auth] Successful login for 'mqtt2'
22-11-15 02:40:20 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.RUNNING
22-11-15 02:40:20 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.DNS_SERVER_IPV6_ERROR/ContextType.DNS_SERVER
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.NO_CURRENT_BACKUP/ContextType.SYSTEM
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.PWNED/ContextType.ADDON
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.IPV4_CONNECTION_PROBLEM/ContextType.SYSTEM
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.FREE_SPACE/ContextType.SYSTEM
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.SUPERVISOR
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.SECURITY/ContextType.CORE
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.DNS_SERVER_FAILED/ContextType.DNS_SERVER
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.check] System checks complete
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
22-11-15 02:40:21 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
22-11-15 02:41:05 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
22-11-15 02:41:07 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-11-15 03:07:46 INFO (MainThread) [supervisor.host.info] Updating local host information
22-11-15 03:07:46 INFO (MainThread) [supervisor.host.services] Updating service information
22-11-15 03:07:46 INFO (MainThread) [supervisor.host.network] Updating local network information
22-11-15 03:07:46 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
22-11-15 03:07:46 INFO (MainThread) [supervisor.host.manager] Host information reload completed
22-11-15 03:11:07 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.RUNNING
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.DNS_SERVER_IPV6_ERROR/ContextType.DNS_SERVER
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.NO_CURRENT_BACKUP/ContextType.SYSTEM
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.PWNED/ContextType.ADDON
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.IPV4_CONNECTION_PROBLEM/ContextType.SYSTEM
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.FREE_SPACE/ContextType.SYSTEM
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.SUPERVISOR
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.SECURITY/ContextType.CORE
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.DNS_SERVER_FAILED/ContextType.DNS_SERVER
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.check] System checks complete
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
22-11-15 03:40:21 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
22-11-15 03:41:07 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
22-11-15 03:41:07 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
22-11-15 03:41:07 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
22-11-15 03:41:07 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-11-15 03:41:08 INFO (MainThread) [supervisor.store] Loading add-ons from store: 69 all - 0 new - 0 remove
22-11-15 03:41:08 INFO (MainThread) [supervisor.store] Loading add-ons from store: 69 all - 0 new - 0 remove
22-11-15 03:41:23 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'mqtt2'
22-11-15 03:41:23 INFO (MainThread) [supervisor.auth] Successful login for 'mqtt2'

SpaceCadetRick commented 1 year ago

I'm having the same issue. Every time I try to access the logs for any add-on I get these errors and my home assistant starts moving really really slowly like it's doing something in the background. If I wait long enough it clears up on it's own but there's nothing logged that indicates what it was working so hard on.

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:42
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 10:02:06 PM (7 occurrences)
Last logged: 10:02:08 PM

Timeout on /addons/a0d7b954_ssh/stats request
Timeout on /addons/6713e36e_rtlamr2mqtt/stats request
Timeout on /addons/core_mosquitto/stats request
Timeout on /addons/core_duckdns/stats request
Timeout on /addons/core_samba/stats request

Logger: homeassistant.components.hassio
Source: components/hassio/__init__.py:1021
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 10:02:08 PM (7 occurrences)
Last logged: 10:02:08 PM

Could not fetch stats for a0d7b954_ssh:
Could not fetch stats for 6713e36e_rtlamr2mqtt:
Could not fetch stats for core_mosquitto:
Could not fetch stats for core_duckdns:
Could not fetch stats for core_samba:

Edit: There seem to be multiple issues currently open that all have the same problem.

https://github.com/home-assistant/core/issues/81276 https://github.com/home-assistant/core/issues/74362

When I try to open a log file from an add-on one of my cpu cores jumps to 100% utilization and which core it is bounces around occationally but then after about 5 minutes it drops back to normal ~15% on each core.

During

After

svadel commented 1 year ago

I have had the same problem for about two months. Sometimes, I can't connect to HASS and at the same time, even Raspberry SSH doesn't work. Usually, this happens to me when I update projects in the ESPHome addon, but not only there.

Version | core-2022.12.8

Host Operating System | Debian GNU/Linux 11 (bullseye)

Update Channel | stable Supervisor Version | supervisor-2022.12.1 Agent Version | 1.4.1 Docker Version | 20.10.22 Disk Total | 109.8 GB Disk Used | 15.5 GB Healthy | true Supported | true Supervisor API | ok Version API | ok

Logger: homeassistant.components.hassio Source: components/hassio/init.py:843 Integration: Home Assistant Supervisor (documentation, issues) First occurred: 18:41:22 (7 occurrences) Last logged: 18:51:54

Could not fetch stats for 5c53de3b_esphome: Could not fetch stats for db21ed7f_portainer: Could not fetch stats for a0d7b954_tailscale:

Logger: homeassistant.components.hassio.handler Source: components/hassio/handler.py:48 Integration: Home Assistant Supervisor (documentation, issues) First occurred: 18:41:22 (11 occurrences) Last logged: 18:51:54

Timeout on /store request Timeout on /os/info request Timeout on /supervisor/info request Timeout on /core/info request Timeout on /addons/5c53de3b_esphome/stats request

Logger: homeassistant.components.hassio Source: components/hassio/init.py:488 Integration: Home Assistant Supervisor (documentation, issues) First occurred: 18:48:10 (1 occurrences) Last logged: 18:48:10

Can't read Supervisor data:

SirDragonClaw commented 1 year ago

Same issue. When this happens I can no longer load the Supervisor logs in the UI and the Add Ons page may become unresponsive. If I try to restart HA, there is a timeout error from the Supervisor.

Lots of errors like this "Failed to to call /addons/core_mosquitto/stats"

Running on a raspberry pi 4 2gb at about 55% memory usage.

svadel commented 1 year ago

Hi! I think that my issue is bad working Linux with a swap file. The kernel accesses too many times to swap file and the CPU stucks up. I solved, the problem by installing zram-swap according to this manual: https://github.com/foundObjects/zram-swap

git clone https://github.com/foundObjects/zram-swap.git cd zram-swap && sudo ./install.sh

If you want to change the config: vi /etc/default/zram-swap

Next step, adding lines to /etc/sysctl.conf: vm.vfs_cache_pressure=500 vm.swappiness=100 vm.dirty_background_ratio=1 vm.dirty_ratio=50

for the better working kernel with ZRAM. sudo sysctl --system or reboot host

I hope it would help you. Sources: https://haydenjames.io/linux-performance-almost-always-add-swap-part2-zram/ https://haydenjames.io/linux-performance-almost-always-add-swap-space/ https://github.com/foundObjects/zram-swap

codemannh commented 1 year ago

That's interesting. I'm not a linux expert by any means, but makes sense. I actually fixed the issue by creating a Linux container and loading everything into Docker containers. Performance is much improved and reliability has been perfect since I did that. Still running on Proxmox, but I ditched the HAOS.

johny-mnemonic commented 1 year ago

@svadel I tried zram swap as well and it definitely improved situation, but it didn't fix it completely. The issue simply seems to be with the performance. If you have too many addons for your HW to handle and most importantly fit into RAM you can start seeing this issue.

nawwman commented 1 year ago

I have the same error:

`Logger: homeassistant.components.hassio.handler Source: components/hassio/handler.py:48 Integration: Home Assistant Supervisor (documentation, issues) First occurred: February 27, 2023 at 15:23:41 (24 occurrences) Last logged: February 27, 2023 at 15:34:15

Timeout on /addons/cebe7a76_hassio_google_drive_backup/stats request
Timeout on /addons/a0d7b954_ssh/stats request
Timeout on /addons/a0d7b954_logviewer/stats request
Timeout on /addons/a0d7b954_nodered/stats request
Timeout on /addons/core_samba/stats request`

`Logger: homeassistant.components.hassio Source: components/hassio/init.py:843 Integration: Home Assistant Supervisor (documentation, issues) First occurred: February 27, 2023 at 15:23:42 (24 occurrences) Last logged: February 27, 2023 at 15:34:15

Could not fetch stats for cebe7a76_hassio_google_drive_backup:
Could not fetch stats for a0d7b954_ssh:
Could not fetch stats for a0d7b954_logviewer:
Could not fetch stats for a0d7b954_nodered:
Could not fetch stats for core_samba:`

System info: `## System Information

version	core-2023.2.5
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.10.7
os_name	Linux
os_version	5.15.84-v7
arch	armv7l
timezone	Europe/Berlin
config_dir	/config

Home Assistant Community Store

Home Assistant Cloud

Home Assistant Supervisor

Dashboards

Recorder

`

bcutter commented 1 year ago

Summary of HA Logfile (contains data from last 62 days):

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:48
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 8. März 2023 um 02:53:18 (111 occurrences)
Last logged: 22:15:34

Timeout on /host/info request
Timeout on /info request
Timeout on /addons/db21ed7f_portainer/stats request
Timeout on /addons/core_mosquitto/stats request
Timeout on /addons/db21ed7f_filebrowser/stats request

Seems to be not complete. Main problem is, that supervisor integration entities regularly switch to unavailable (triggering my monitoring implemented according to https://github.com/jazzyisj/unavailable-entities-sensor / https://community.home-assistant.io/t/unavailable-unknown-entity-monitoring-template-sensor/147618).

Some are very intensively creating this issue (https://github.com/alexbelgium/hassio-addons/issues/745), for other addons this happens quite rarely.

At the end... the supervisor itself "feels" quite unreliable because of this.

HA OS 9.4
HA Core 2022.12.9
HA Supervisor 2023.3.1

issue-triage-workflows[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

bcutter commented 1 year ago

Not solved.

gigatexel commented 1 year ago

Same here, for a long time now.

bcutter commented 1 year ago

I think it's a Supervisor issue and therefore in the wrong repo here by the way.

robertvorthman commented 1 year ago

For those that have not solved the issue, try plugging your HA machine into a computer monitor. I found an error message that did not appear in the logs that helped me track down the cause of my Home Assistant errors/slow downs/add-ons failing.

I noticed that when Home Assistant started running slow, UI failing, add-ons stopping, that an error would show on the monitor. For me, it was "cifs: vfs: "" has not responded in 180 seconds. reconnecting"" that would correlate with Home Assistant slowing down and add-ons breaking. I deleted that SMB share from my HA storage settings and the issue has not reoccured.

bcutter commented 1 year ago

Which SMB share?

Why should this be the root cause and not just another symptom of a slow system?

Sounds a bit too easy to be true :-)

Maybe a load/system pressure thing.

gigatexel commented 1 year ago

I don't have any shares in Network storage, so I doubt this is the root cause.

robertvorthman commented 1 year ago

Which SMB share?

Why should this be the root cause and not just another symptom of a slow system?

Sounds a bit too easy to be true :-)

Maybe a load/system pressure thing.

It was a standard network SMB share to a QNAP NAS that I configured Home Assistant to use as network storage for backups. Besides deleting the network storage from HA settings, I also updated all my add-ons, like the Unifi add-on, so I'm not sure exactly what solved my issue, but I just wanted to let others know that plugging in their machine to a monitor can yield more info.

issue-triage-workflows[bot] commented 10 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

bcutter commented 10 months ago

Still exists

agners commented 10 months ago

This is likely a Supervisor issue indeed.

Can someone share the logs of Supervisor from the same time when the issue is reported by Home Assistant Core itself? I wonder if the Supervisor logs have an indication why stats can't be loaded.

bcutter commented 10 months ago

This is likely a Supervisor issue indeed.

Can someone share the logs of Supervisor from the same time when the issue is reported by Home Assistant Core itself? I wonder if the Supervisor logs have an indication why stats can't be loaded.

Last entry:

HA Core

2023-11-07 20:36:32.144 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/db21ed7f_filebrowser/stats request
2023-11-07 20:36:32.148 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/db21ed7f_portainer/stats request
2023-11-07 20:36:32.152 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/a0d7b954_vscode/stats request
2023-11-07 20:36:32.157 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_configurator/stats request

HA Supervisor

23-11-07 20:36:32 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)

I'm not sure if this is a coincidence. Because: Supervisor logs that same message much more and it does not match the timeout on ... stats request log error of HA Core. Example:

More of HA Supervisor:

23-11-07 20:15:58 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 20:21:05 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 20:26:12 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 20:31:19 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 20:36:32 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 20:41:39 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 20:46:46 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 20:51:53 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 20:57:00 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:02:07 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:07:14 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:12:21 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:17:28 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:22:35 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:27:42 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:32:49 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:37:56 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:43:03 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:48:10 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:53:17 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 21:58:24 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
23-11-07 22:03:31 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)

And for all those entries (except the ones from 20:36:32) there are no timeout on ... stats request log errors in HA Core.

Edit: New entries in HA Core:

2023-11-07 22:08:44.057 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/db21ed7f_portainer/stats request
2023-11-07 22:08:44.063 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/a0d7b954_vscode/stats request
2023-11-07 22:08:44.068 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/db21ed7f_filebrowser/stats request

Content in HA Supervisor around that time:

23-11-07 22:07:35 INFO (SyncWorker_5) [supervisor.docker.manager] Stopping addon_core_samba application
23-11-07 22:07:40 INFO (SyncWorker_5) [supervisor.docker.manager] Cleaning addon_core_samba application
23-11-07 22:07:41 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on homeassistant/aarch64-addon-samba with version 10.0.2
23-11-07 22:07:42 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from core_samba
23-11-07 22:09:24 INFO (SyncWorker_6) [supervisor.docker.manager] Stopping addon_core_samba application
23-11-07 22:09:29 INFO (SyncWorker_6) [supervisor.docker.manager] Cleaning addon_core_samba application

(I just reconfigured the SAMBA share addon and restarted it two times)

I don't think this is a "hot track" -.-

issue-triage-workflows[bot] commented 7 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

bcutter commented 7 months ago

Existing and annoying. Bye stale label, unfortunately.

zavjah commented 6 months ago

same issue at my side, are there any updates on this?

I do have an network storage set up for backups, but it not SMB but NFS. Nevertheless, I'll delete that and see what happens.

issue-triage-workflows[bot] commented 3 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core