search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
70.91k stars 29.56k forks source link

HAOS Core: "Fatal Python error: Segmentation fault" in RaspberryPi4 after 2024.02 upgrade related to threading #110538

Closed diamant-x closed 2 months ago

diamant-x commented 6 months ago

The problem

After 2024.02 upgrade, my HA randomly restarts on its own at least 2-3 times per day. Haven't found a pattern to it. System resources usage seems normal. HA created a .fault log file which I'm attaching from the last occurence today. I'm also adding the log file from before and after the restart; can't see anything odd.

Based on the error seems to be some issue when using the multiprocessing in some core function.

I'm happy to provide more details. I installed HA on a new SD card back on early December and was running just fine until the previous version, so shouldn't be tear-down related AFAIK.

What version of Home Assistant Core has the issue?

core-2024.2.1

What was the last working version of Home Assistant Core?

core-2024.1.6

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Unknown (iperf3?)

Link to integration documentation on our website

Unknown

Diagnostics information

home-assistant.log

I'm changing the file extensions so that it can be uploaded. home-assistant.1.log

home-assistant.fault.log

Example YAML snippet

N/A

Anything in the logs that might be useful for us?

Attached above the fault log file.

Additional information

System Information

version core-2024.2.1
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.1
os_name Linux
os_version 6.1.63-haos-raspi
arch aarch64
timezone Europe/Madrid
config_dir /config
Home Assistant Community Store GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 5000 Installed Version | 1.34.0 Stage | running Available Repositories | 1396 Downloaded Repositories | 27
Home Assistant Cloud logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Home Assistant Supervisor host_os | Home Assistant OS 11.5 -- | -- update_channel | stable supervisor_version | supervisor-2024.01.1 agent_version | 1.6.0 docker_version | 24.0.7 disk_total | 116.5 GB disk_used | 77.6 GB healthy | true supported | true board | rpi4-64 supervisor_api | ok version_api | ok installed_addons | AdGuard Home (5.0.3), ESPHome (2023.12.9), FTP (5.0.1), Firefly iii (6.1.9), Firefly iii Data Importer (1.4.3), Home Assistant Google Drive Backup (0.112.1), InfluxDB2 (0.1.0.0), WireGuard (0.10.2), Nginx Proxy Manager (1.0.1), Samba share (12.3.0), Prowlarr NAS (nightly-1.13.3.4259-ls111), Vaultwarden (Bitwarden) (0.21.1), chrony (3.0.1), motionEye (0.20.0), qBittorrent (4.6.3-4), Sonarr (4.0.1.1096), Radarr (5.2.6.8376), Lidarr NAS (2.0.7.3849), Plex Media Server (3.4.1), MariaDB (2.6.1), Bazarr NAS (1.4.1), DOODS2 (1), Castsponsorskip (0.7.2), phpMyAdmin (0.9.1), Changedetection.io (0.45.14), Browserless Chrome (2.0.0), File editor (5.8.0)
Dashboards dashboards | 7 -- | -- resources | 17 views | 17 mode | storage
Recorder oldest_recorder_run | 17 de diciembre de 2023, 12:29 -- | -- current_recorder_run | 14 de febrero de 2024, 04:29 estimated_db_size | 4104.64 MiB database_engine | mysql database_version | 10.6.12
diamant-x commented 6 months ago

This is too unstable, here is a chart showing how right after the update is installed the restarts start occurring.

image

If I look closer, latest days all reboots happen exactly at around 4hours 10 minutes sin last restart: image

With same error stack: home-assistant.fault..log

slovdahl commented 6 months ago

I'm seeing something similar since upgrading from 2024.1 to 2024.2.1 (home-assistant.log.fault.txt). Raspberry Pi 4, Ubuntu 20.04 64-bit, using Home Assistant Container.

$ uname -r -v -i
5.4.0-1101-raspi #113-Ubuntu SMP PREEMPT Fri Jan 12 18:10:28 UTC 2024 aarch64
slovdahl commented 6 months ago

Might be fixed in 2024.2.2: https://github.com/home-assistant/core/issues/110464#issuecomment-1949023744

diamant-x commented 6 months ago

Might be fixed in 2024.2.2: #110464 (comment)

That one mentions python 3.11, but my fault stack strace only shows python 3.12. let's see if we're lucky and it gets fixed too but could potentially not be the same.

diamant-x commented 6 months ago

I confirm is still an issue in 2024.2.2. I can't believe is just a couple of us?

Versión core-2024.2.2
Tipo de instalación Home Assistant OS
Desarrollo false
Supervisor true
Docker true
Usuario root
Entorno virtual false
Versión de Python 3.12.1
Familia de Sistema Operativo Linux
Versión del Sistema Operativo 6.1.63-haos-raspi
Arquitectura de CPU aarch64
DutchCruZer4 commented 6 months ago

I confirm is still an issue in 2024.2.2. I can't believe is just a couple of us? Versión core-2024.2.2 Tipo de instalación Home Assistant OS Desarrollo false Supervisor true Docker true Usuario root Entorno virtual false Versión de Python 3.12.1 Familia de Sistema Operativo Linux Versión del Sistema Operativo 6.1.63-haos-raspi Arquitectura de CPU aarch64

Nope not a couple of us. I have the same issue.

I'm running on Proxmox VE 8.1

**Fatal Python error: Segmentation fault

Thread 0x00007f1b3d0ffb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3d71fb30 (most recent call first): File "/usr/local/lib/python3.12/threading.py", line 359 in wait File "/usr/local/lib/python3.12/threading.py", line 655 in wait File "/usr/src/homeassistant/homeassistant/components/stream/init.py", line 410 in _run_worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3de3fb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3ee46b30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3f133b30 (most recent call first): File "/usr/local/lib/python3.12/threading.py", line 355 in wait File "/usr/local/lib/python3.12/queue.py", line 171 in get File "/usr/local/lib/python3.12/site-packages/watchdog/observers/api.py", line 367 in dispatch_events File "/usr/local/lib/python3.12/site-packages/watchdog/observers/api.py", line 202 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3f246b30 (most recent call first): File "/usr/local/lib/python3.12/threading.py", line 355 in wait File "/usr/local/lib/python3.12/site-packages/watchdog/utils/delayed_queue.py", line 51 in get File "/usr/local/lib/python3.12/site-packages/watchdog/observers/inotify_buffer.py", line 41 in read_event File "/usr/local/lib/python3.12/site-packages/watchdog/observers/inotify.py", line 128 in queue_events File "/usr/local/lib/python3.12/site-packages/watchdog/observers/api.py", line 145 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4186cb30 (most recent call first): File "/usr/local/lib/python3.12/site-packages/watchdog/observers/inotify_c.py", line 304 in read_events File "/usr/local/lib/python3.12/site-packages/watchdog/observers/inotify_buffer.py", line 89 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b43c7fb30 (most recent call first): File "/usr/local/lib/python3.12/site-packages/scapy/supersocket.py", line 264 in select File "/usr/local/lib/python3.12/site-packages/scapy/sendrecv.py", line 1219 in _run File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4537fb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b40c59b30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b40d6cb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4287fb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b46496b30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b465a9b30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b466bcb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b467cfb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b480efb30 (most recent call first): File "/usr/local/lib/python3.12/threading.py", line 359 in wait File "/usr/local/lib/python3.12/queue.py", line 180 in get File "/usr/local/lib/python3.12/site-packages/aiosqlite/core.py", line 99 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5bb5eb30 (most recent call first): File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 579 in run_once File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 540 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b48f0fb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4a519b30 (most recent call first): File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 579 in run_once File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 540 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4a919b30 (most recent call first): File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 579 in run_once File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 540 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4f1cfb30 (most recent call first): File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 579 in run_once File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 540 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5588fb30 (most recent call first): File "/usr/local/lib/python3.12/threading.py", line 359 in wait File "/usr/local/lib/python3.12/threading.py", line 655 in wait File "/usr/local/lib/python3.12/site-packages/pychromecast/discovery.py", line 304 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5608fb30 (most recent call first): File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4bd2cb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4c43fb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4e75cb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5196fb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/src/homeassistant/homeassistant/components/recorder/executor.py", line 17 in _worker_with_shutdown_hook File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b51f8fb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/src/homeassistant/homeassistant/components/recorder/executor.py", line 17 in _worker_with_shutdown_hook File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b595afb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b2ccb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b3e3b30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b4fab30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b611b30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b728b30 (most recent call first): File "/usr/local/lib/python3.12/site-packages/paho/mqtt/client.py", line 1150 in _loop File "/usr/local/lib/python3.12/site-packages/paho/mqtt/client.py", line 1756 in loop_forever File "/usr/local/lib/python3.12/site-packages/paho/mqtt/client.py", line 3591 in _thread_main File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b83fb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5bd75b30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5be8cb30 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Current thread 0x00007f1b5c69fb30 (most recent call first): File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 152 in _orjson_default_encoder File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 180 in save_json File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 364 in _write_data File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b67d5db38 (most recent call first): File "/usr/local/lib/python3.12/selectors.py", line 462 in select File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1936 in _run_once File "/usr/local/lib/python3.12/asyncio/base_events.py", line 638 in run_forever File "/config/custom_components/ble_monitor/init.py", line 702 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b67f74b38 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b78c5ab38 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b7a76fb38 (most recent call first): File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 872 in _run_event_loop File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 778 in _run File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 698 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b845ffb38 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b84ffbb38 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b850ffb38 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b859ffb38 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b87b97b38 (most recent call first): File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b87cbfb38 (most recent call first): File "/usr/local/lib/python3.12/logging/handlers.py", line 1527 in dequeue File "/usr/local/lib/python3.12/logging/handlers.py", line 1578 in _monitor File "/usr/local/lib/python3.12/threading.py", line 1010 in run File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b8cc01b08 (most recent call first): File "/usr/local/lib/python3.12/selectors.py", line 462 in select File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1936 in _run_once File "/usr/local/lib/python3.12/asyncio/base_events.py", line 638 in run_forever File "/usr/local/lib/python3.12/asyncio/base_events.py", line 671 in run_until_complete File "/usr/src/homeassistant/homeassistant/runner.py", line 188 in run File "/usr/src/homeassistant/homeassistant/main.py", line 209 in main File "/usr/src/homeassistant/homeassistant/main.py", line 221 in File "", line 88 in _run_code File "", line 198 in _run_module_as_main**

diamant-x commented 6 months ago

Not sure if could be related, but the AEMET weather integration started failing at a similar moment:

image

Not sure if there's any specific way to point to AEMET developer to check if this might be related?

DutchCruZer4 commented 6 months ago

Not sure if could be related, but the AEMET weather integration started failing at a similar moment:

image

Not sure if there's any specific way to point to AEMET developer to check if this might be related?

Found my problem. It was the hacs watchman plugin what got screwed. Disabled it and everything is fine now.

rds76 commented 6 months ago

I closed #110464 but after 4days it crashed again, same symptoms - Fatal Python error: Segmentation fault, most recent version 2024.2.2 with updated orjson. Looks like Shelly is always involved

Current thread 0xb7d606c0 (most recent call first):
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 72 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1528 in json_fragment
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 70 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/json.py", line 48 in json_encoder_default
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1136 in json_fragment
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 70 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/websocket_api/messages.py", line 138 in _partial_cached_event_message
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/websocket_api/messages.py", line 122 in cached_event_message
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/websocket_api/commands.py", line 132 in _forward_events_unconditional
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1253 in async_fire
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1924 in async_set
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/entity.py", line 1172 in _async_write_ha_state
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/entity.py", line 989 in async_write_ha_state
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/shelly/entity.py", line 341 in _update_callback
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 183 in async_update_listeners
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 439 in async_set_updated_data
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/shelly/coordinator.py", line 336 in _async_handle_update
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/aioshelly/block_device/device.py", line 248 in _update_s
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/aioshelly/block_device/device.py", line 196 in _coap_message_received
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/aioshelly/block_device/coap.py", line 185 in datagram_received
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 1169 in _read_ready
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80 in _run
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922 in _run_once
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607 in run_forever
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640 in run_until_complete
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/runner.py", line 188 in run
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/__main__.py", line 209 in main
  File "/opt/venv/homeassistant/bin/hass", line 8 in <module>
diamant-x commented 6 months ago

Found my problem. It was the hacs watchman plugin what got screwed. Disabled it and everything is fine now.

How did you figured out which one it was? I tried disabling a couple of HACS which haven't been updated in some time (so potentially not supporting Python 3.12) but without success.

DutchCruZer4 commented 6 months ago

Found my problem. It was the hacs watchman plugin what got screwed. Disabled it and everything is fine now.

How did you figured out which one it was?

I tried disabling a couple of HACS which haven't been updated in some time (so potentially not supporting Python 3.12) but without success.

I had an ! on my device_tracker.yaml file. I checked it and it gave me an error on the file with a line couldn't be empty or only had special characters. I first disabled my ble_tracker and rebooted. Same fault. Then I disabled the watchman plugin and rebooted and fault was gone. It also had a lot of devices in the log and errors that devices where not found.

I think it brought my Home Assistant install in a constant error loop and that it got to much so the core got rebooted every now and then.

diamant-x commented 6 months ago

Ok, now is clear something is wrong. I rebooted into 'safe mode' HA to ensure if its was a HACS, or something else that was messing only to find that it keeps crashing at around ~4h:10m just as shown in OP.

Now I think I know what it is: iperf integration, which I've set it such scan interval to be that one. Not a coincidence the last log message before crash is precisely a server unavailable. Nothing new that shouldn't crash HA.

home-assistant.1.log home-assistant.fault.log

diamant-x commented 6 months ago

Hey there @rohankapoorcom , mind taking a look at this issue as it has been labeled with an integration (iperf3) you are listed as a code owner for? Thanks!

rds76 commented 6 months ago

Forget it is iPerf. Mine is still crashing and I'm not using iperf at all. It is definitelly core issue!

diamant-x commented 6 months ago

Forget it is iPerf. Mine is still crashing and I'm not using iperf at all. It is definitelly core issue!

Crap, thanks for the update, will revert the ticket title and integration. Something that bothers me: in my last fault that shows the segmentation fault, it mentions some custom integrations that shouldn't have been enabled as I was in safe boot mode. I'll try to remove them from the custom_components folder and try again into safeboot to see if it has any effects.

Another work action I took, I tried to understand if Python 3.12 had any main changes regarding threading and could find there has been important changes https://peps.python.org/pep-0684/ Not sure if HA devs already announced an impact due to this?

rds76 commented 6 months ago

Well, I'm still on Python 3.11, 1st thoughts was that, but I see segfaults on 3.12 here.. Must be related to 2024.02 as with previous release I had weeks of uptime with no issues. Such an error is mostly related to stack memory, buffer overflow, etc., not threading.

FreddyX078 commented 6 months ago

Add me to the list, same problem here. Using 2024.2.2 with Python 3.12. HA (HAOS on Beelink mini) restarts aprox 10 - 20 times a day :-(

MemTest ran for a couple of hours, no problem detected. Booted into safe mode but same crash happend.

Error from supervisor: 24-02-22 11:36:27 WARNING (MainThread) [supervisor.homeassistant.websocket] Connection is closed 24-02-22 11:36:30 WARNING (MainThread) [supervisor.homeassistant.core] Watchdog found Home Assistant failed, restarting...

rds76 commented 6 months ago

This is my commont part where it ends with segfault (3 crashed in recent 24h)

  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 72 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1528 in json_fragment
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 70 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/json.py", line 48 in json_encoder_default

I suspect orjson https://github.com/ijl/orjson/issues/458

DavidBuchanan314 commented 6 months ago

afaik https://github.com/ijl/orjson/issues/458 has been an issue since day 1, if you're seeing "new" crashes I think it's more likely to be related to https://github.com/ijl/orjson/pull/457

Zaczero commented 6 months ago

This is most likely due to https://github.com/ijl/orjson/issues/452. The crashes affect all orjson versions starting from 3.9.11. The crash rate was reduced in 3.9.14, but it's apparently still present. The HOC seems to use 3.9.14.

Warning! Personal Opinion: I migrated from orjson to msgspec which has comparable performance and a similar set of features. I really don't like how the whole situation was/is handled by the orjson maintainers. They seem to prioritize performance over stability too much, which is just not acceptable for me. Also, they don't seem to listen to feedback, and response times are meh. orjson is also currently missing a security policy which is yet another red flag.

FreddyX078 commented 6 months ago

An short update: after deleting all refresh-tokes, system is up for 8 hours. Hopefuly it will survive the night.

diamant-x commented 6 months ago

custom integrations that shouldn't have been enabled as I was in safe boot mode. I'll try to remove them from the custom_components folder and try again into safeboot to see if it has any effects.

An update from my side. I did this and the system didn't reboot in a full day. I re-added the custom_components folder and performed a normal restart again and the system now is not crashing. I hope it remains like that. I would recommend all of you to try:

  1. Rename the 'custom_components' folder to 'temp_custom_components'
  2. Reboot into Safe Boot mode.
  3. See if it crashes or not.
  4. Rename back from step 1
  5. Reboot normal to reenable customization.

I think this might merit for a new bug ticket:

diamant-x commented 6 months ago

the system now is not crashing.

It crashed again. core 2024.2.3

home-assistant.fault.log

System Information

version core-2024.2.3
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.1
os_name Linux
os_version 6.1.63-haos-raspi
arch aarch64
config_dir /config
Home Assistant Community Store GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 5000 Installed Version | 1.34.0 Stage | running Available Repositories | 1405 Downloaded Repositories | 27
Home Assistant Cloud logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Home Assistant Supervisor host_os | Home Assistant OS 11.5 -- | -- update_channel | stable supervisor_version | supervisor-2024.02.0 agent_version | 1.6.0 docker_version | 24.0.7 disk_total | 116.5 GB disk_used | 75.8 GB healthy | true supported | true board | rpi4-64 supervisor_api | ok version_api | ok installed_addons | AdGuard Home (5.0.3), MariaDB (2.6.1)
Dashboards dashboards | 7 -- | -- resources | 17 views | 17 mode | storage
Recorder oldest_recorder_run | 30 de diciembre de 2023, 10:35 -- | -- current_recorder_run | 26 de febrero de 2024, 15:02 estimated_db_size | 4023.14 MiB database_engine | mysql database_version | 10.6.12
diamant-x commented 5 months ago

Still an issue in core-2024.2.5

home-assistant.fault.log

I'm wondering, could it be a Database related issue? I'm using the addon MariaDB Current version: 2.6.1.

System Information

version core-2024.2.5
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.1
os_name Linux
os_version 6.1.73-haos-raspi
arch aarch64
config_dir /config
Home Assistant Community Store GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 4851 Installed Version | 1.34.0 Stage | running Available Repositories | 1405 Downloaded Repositories | 27
Home Assistant Cloud logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Home Assistant Supervisor host_os | Home Assistant OS 12.0 -- | -- update_channel | stable supervisor_version | supervisor-2024.02.1 agent_version | 1.6.0 docker_version | 24.0.7 disk_total | 116.5 GB disk_used | 76.5 GB healthy | true supported | true board | rpi4-64 supervisor_api | ok version_api | ok
Dashboards dashboards | 7 -- | -- resources | 17 views | 17 mode | storage
Recorder oldest_recorder_run | 4 de enero de 2024, 16:33 -- | -- current_recorder_run | 4 de marzo de 2024, 14:21 estimated_db_size | 4241.77 MiB database_engine | mysql database_version | 10.6.12
issue-triage-workflows[bot] commented 2 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.