search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
73.78k stars 30.88k forks source link

Crashing everyday #43155

Closed RavD666 closed 3 years ago

RavD666 commented 4 years ago

home assistant crashing everyday# The problem

Hass.io keeps crashing everyday,I can see it on my router. Have disconnect the power to boot it back up. Its only started doing it since the last few updates.

How can I get the logs to find the issue

Environment

Problem-relevant configuration.yaml

Traceback/Error logs

Additional information

Salamandar commented 3 years ago

It started crashing regularly again. No sqlite errors in the logs this time (but a docker image keeps trying to start).

igosoft commented 3 years ago

@Salamandar I downgraded as it was suggested to 5.5 ha os update --version=5.5 added recorder configuration to HA recorder: purge_keep_days: 5 commit_interval: 60 exclude: domains:

and moved data do usb disk drive using datactl move /dev/sda

and so far it works fine. It looks like recorder settings will have some influence on behaviour. However I'll be playing with the newest ha os version during incoming weekend as well to conform that

muzzak123 commented 3 years ago

I've been on HASSOS 5.4 and Home Assistant 2021.2.3 for nearly a week with not one crash and no issues with anything. Still running lots of different integrations and addons .. Samba share (9.3.0), Duck DNS (1.12.5), ESPHome (1.16.2), AdGuard Home (3.0.0), Home Assistant Google Drive Backup (0.103.1), Mosquitto broker (5.1), MariaDB (2.2.1), Visual Studio Code (3.1.0), Node-RED (8.1.2), TasmoBackup (1.04.08), Grafana (6.1.2) All working fine. I'll wait to see what others say about HASSOS 5.12. Maybe some joy in there ?

sitanshshah commented 3 years ago

Hey All,

Relative newbie here - so go easy on me ;)

Background: After reading all the posts on this thread, I can also report my HA is crashing twice aday now. Started to be every few days, then once a day, now multiple times aday. I'm not a pro so not sure which logs would be helpful, but my spec is :

pi 4 8GB (2 weeks old) 128 gb SD card latest version of HA Also had Fan Shim, I used to toggle on and off if the CPU temp was over 55c

What have I done about it ? I have literally rolled back to HAASOD 5.4, but now my Pii wont boot up, as reported by ben8p

Questions

  1. Is my only option now to reinstall and start again ? I have a back up , but just exploring my options ?
  2. is there away to install 5.4 version directly , rather than the latest ? yeah I know newbie question ;)
  3. if not, once I'm back up and running, what is the best procedure to roll back to a safe version ?

Thank you in advance - hopefully I can give something back to the community in the future

Cheers

djdemer commented 3 years ago

I share what your going through. Your excited, learning, experimenting, and testing a new tool and become frustrated that the whole thing crashes and wonder what about other users, what are they doing different? I too backed up my Snapshots and reverted to 5.4 and it caused my boot up to fail. I then re-etched my system to 5.9, booted up, restored my Snapshots and started over with several integrations. Its been running fine for one week now. I am happy operating at this level of code for the time being and continue progress with my installation.

sitanshshah commented 3 years ago

thanks djdemer - and more newbie questions (wonder how long I can play the newbie card LOL)

question - when you say you re-etched my system to 5.9 what do you mean ?

Also I've been using the link to github on the guide here : https://www.home-assistant.io/installation/raspberrypi It is for 5.12 , am I reading that right ?

Should I be on a different version ? if so how do I get the img for specific versions ?

djdemer commented 3 years ago

I used the Full installation Home Assistant method your following:

This OS image below is tarred into a a packed file that can be restored using a program called Etcher written by Balena.

https://github.com/home-assistant/operating-system/releases/tag/5.9

sitanshshah commented 3 years ago

Thanks for the link - I'm still feeling my way around on github. I've restored to Home Assistant OS 5.9 - I'll report back in a few days

sitanshshah commented 3 years ago

Quick update from me.

installed new image to 5.9, and 5.5 - unfortunately I still saw crashes with both versions. I see some folks have successfully installed 5.4, but for me I cannot boot up. Those folks with 5.4 - how did you resolve the boot issue ?

I'm also wondering if I've installed something (addon) thats causing the crashes, rather than the OS. I'm seeing some errors in my logs, but sure if they are related.

Any advice

Sitansh

muzzak123 commented 3 years ago

Hi @sitanshshah, I never had a boot issue to resolve. Just rolled back to 5.4 and away it went. @frenck - Just wondering if this issue should be changed from closed back to open as it doesn't appear @RavD666 has provided any useful answers as to how it was resolved ? And although HASSOS 5.4/5 is a workaround, it certainly is not truly resolved for the many people commenting here.

SebSkarp commented 3 years ago

Hi all I´m very new with HA, but I absolutely love it! Long story short, I too experienced my Rpi with HA freezing multiple times a day, with the Rpi CPU temp being very high but the entire system just froze on me. Could not access, samba, ping or anything. Only solution was to unplug power to reboot. Worked fine for some time then the same all over again. This seemed to start with the latest 5.12 OS update, for me at least I had no issues before. Nothing in the logs that I could see, nor anything pointing me to the cause. Got a brand new SD card, reinstalled multiple times and used snapshot. What turned out to work, for me at least, was to flash the sd card with OS 5.1 version, then use partial snapshot (excluding Home assistant checkbox in the top) to restore. I removed some addons (Aircast, CEC scanner Log viewer and spotify connect), if this helped or not I´m not sure of.

My rpi has now been going strong with no issues of freeze for little over 24 hrs. I´m still prepared for the issues to appear again, but fingers crossed it won´t. Don´t know if there is something in v5.12 that caused this for me, but at least with 5.10 it seems to be working again.

sitanshshah commented 3 years ago

@muzzak123 I would agree with you - its obviously a wide spread issue. 5.4 would cause my setup not to boot up - looks like thats a known issue too.

@SebSkarp - thanks for sharing your experience - good to know I'm not the only newbie to be feeling this pain ;)

@Everyone else I've rolled back to 5.3 and its been working for just under 24hours without and hiccups - famous last words eh LOL.

HumanSkunk commented 3 years ago

Just noticed this issue thread as I had been partaking in the one mentioned above #1119. The workaround I have is using 5.3. Has worked perfectly for over a month since trying 5.11 with no dice. A number of people have provided logs on that thread, but there is a lot of information on there. One of the guys generated a spreadsheet linked here where we have been adding details around the builds to see if there is a correlation. I personally cant see one, https://docs.google.com/spreadsheets/d/1iHTVvaNlTUqwFUgsUhUNws2Sw115INIx5ChEgTnIfoc/edit#gid=0 Sorry I cant be of more help but running 5.3 has meant I didnt need to use relays or other automations to attempt to reboot the Pi.

VolkerLaude commented 3 years ago

Same here - I bought a RPI4 in November last year. The performance was impressive. But my home assistant was getting less reliable day by day. I'm keeping the OS, HA versions, add-ons and also the equipments to the latest available versions. SD-Card and power supply are as required! The HA is unstable with both RPI4 versions 64 or 32 bit. I'm quite confident this is not related to the OS level, because since my HA is running on the old and slow RPI3 now and all is rock solid again. OS 5.12 and Core 2021.2.3 is running without any issues for more then 4 days now. Since the new RPI has a mini HDMI onyl, it is not so easy to plug a external monitor. It is just awful seeing people requesting to get logs or screenshot and keep on telling you are the only one on earth with issues like this. It was great to use Homeassistant and joined into it by chance, may I'll for leave for the same reason.

pataar commented 3 years ago

Same issue occurred to me. Logs were telling about a recorder timeout. So I switched from SQLite to MariaDB as backend, and HA hasn't crashed since then. Perhaps this helps other people as well

edelans commented 3 years ago

same here. HA is very unstable, crashes every day. a manual reboot (remove power plugs) works. I use a RPi.

Where can I find the logs to be helpful here ?

More specifically :

image

muzzak123 commented 3 years ago

@pataar I switched to MariaDB early on in my testing to try to see if it made any difference but it didn't. help. Still crashed several times a day.

There's another link of people experiencing the same issue here https://github.com/home-assistant/operating-system/issues/1232. It seems there are many people experiencing the same or similar issue on RPIs. There are many disjointed records in all these different threads. Even this thread has been closed with out any real solution provided and yet people are continuing to contribute.

I think HA is an awesome product and the result of a lot of hard work, by a lot of people, over a lot of years. It is a shame to see it's reputation brought down by this issue, especially as an RPI is a cheap and easy base hardware for home automation.

I feel if this issue is ever going to be resolved adequately there needs to be a coordinate approach to defining the problem and capturing the problem details in a central location. There is a lot of information scattered amongst these various threads but it is hard to see the big picture when it is so disjointed. Some advice from others as to the best way to collate this data would be great. The spreadsheet @HumanSkunk has linked is a start ie So far the issue seems to be:

There also needs to be someone or group on the development team who is willing to work with us users to try to resolve this issue and provide possible solutions or fixes to try. Without this developer buy-in we can't make any progress.

I am more than willing to contribute in anyway I can and have done some of this already previously in this thread.

soundslikeluke commented 3 years ago

I am having the exact same issue I guess I just need to what to see if updates will help. I thought it was an issue with my ssd but turns out I am not alone. I am running a Pi 3B+ on the latest version of everything 64 bit on a ssd. It is happening multiple times per day taking a long time to reboot.

agners commented 3 years ago

@muzzak123 thanks for summarizing the situation, I do agree with your assessment. The problem is with this kind of problems, if none of the core devs hit it, it usually takes longer and more coordination to get it fixed :(

There also needs to be someone or group on the development team who is willing to work with us users to try to resolve this issue and provide possible solutions or fixes to try.

I am tracking the problem off and on, and I am trying to help isolate the problem in various issues.

Please let's move the discussion to the appropriate issues in the OS repository, such as https://github.com/home-assistant/operating-system/issues/1119.

SebSkarp commented 3 years ago

Update from my end. Since this was going on and on even after what I thought fixed the crash/freeze, I got fed up and got myself an Intel Nuc. Best ever. No issues at all. So I’m guessing it came down to memory issues. Maybe faulty device. But more likely my Rpi3b+ just didn’t have enough power. Sad but true.

bschatzow commented 3 years ago

There are directions on how to get to the previous logs. Check out issue 1119. Without the logs, the is no chance how to fix this. Briefly you need to first setup ssh on port 22222. From there there are ample directions on what to do. Send me a on if you can't find information or need clarification.

eizemazal commented 3 years ago

The issue is closed, but does not look to be resolved, the logs that we posted did not help yet.

Seems I have a workaround, my Home Assistant install is running for 24 h for now for the first time on the same hardware. HA is the latest version. Instead of installing HASSOS, I burnt Ubuntu 64 bit image for Raspberry Pi 4 using Balena Etcher, booted Pi and logged in by SSH (I neither have monitor nor keyboard), installed Docker as described here https://docs.docker.com/engine/install/ubuntu/, and then installed home assistant container as described here https://www.home-assistant.io/installation/linux

There is no support for addons, but they are mostly either useless in this installation (like MariaDB or DHCP server, easier to install and configure in the host system if you need), or can be installed manually as Docker containers. HACS works without any issues, I installed it by pasting just one command, and used it.

(Just BTW - still curious why HA offers its own OS which does not look to have any advantages over decent Linux distributions. As a newbie to HA but experienced Linux user, I had a pretty unpleasant experience with a system that tries to hide host OS from me. Other people also had problems getting host OS logs because of this strange architecture.)

Salamandar commented 3 years ago

AFAIK, I fixed the issue by reading the systemd logs (from the sd card, on a x86 host) and resolving the misc issues that were just logging a lot.

eizemazal commented 3 years ago

AFAIK, I fixed the issue by reading the systemd logs (from the sd card, on a x86 host) and resolving the misc issues that were just logging a lot.

So you mean the problem was just because some processes dumped too much data on the log? I have read my logs, there was also a lot of noise, and the crash was preceded by a burst of messages. But this was not expected to crash the entire system...

Salamandar commented 3 years ago

@au1985 Yes. At least that's what I think. This would be better to have the dmesg but I only have the journalctl.

Every time Hass crashed and I retrieved the logs, there was something logging a lot, like :

And every time, I fixed the cause of those logs and the crashes stopped.

eizemazal commented 3 years ago

@Salamandar an interesting observation. But it is not clear to me how crash of logger daemon process can lead to a crash of the entire system...

Salamandar commented 3 years ago

@Salamandar an interesting observation. But it is not clear to me how crash of logger daemon process can lead to a crash of the entire system...

Neither it is to me ! The only thing I see is that the logging is done on a sd card. That may be a reason why Armbian mounts a ramdisk on /var/log…

redd-bbeard commented 3 years ago

I’ve got this same problem. I have done a lot of troubleshooting, I even rebuilt my whole system from the ground up with a new SD card, an A2 card but it seems to be completely random.. I have had a monitor and keyboard connected for about a week and it seems that when the system crashes I lose connection in every way, samba, ssh, web GUI, cloud web GUI, voice commands, even including the terminal on the device with a physical keyboard and HDMI, HASSIO is completely unresponsive the little cursor in the terminal window stops flashing. I’ve also noticed the green light on the front of my PI stops blinking too. The only way to resolve is a full host power cycle. I even implemented a script to tell the system to ping my routers IP and if it loses connection for a minute to run a shell command that saves the log as a separate file and restart the host but even that doesn’t work when the freeze happens. Even if I take my whole network down manually upon HASSIO freezing and wait a few minutes it will not run the automation. Everything is completely frozen with no logs to troubleshoot.

eiiot commented 3 years ago

Google Cast was the issue for me. Disabled it, and I've been stable for 2 days, when I was crashing every hour or so.

GilDev commented 3 years ago

Same problem here…

Cuxx commented 3 years ago

Google Cast was the issue for me. Disabled it, and I've been stable for 2 days, when I was crashing every hour or so.

Same for me. Thanks for the Info. My Pi is running for 2+ days now (never worked so long without a crash).

redd-bbeard commented 3 years ago

Google Cast was the issue for me. Disabled it, and I've been stable for 2 days, when I was crashing every hour or so.

Same for me. Thanks for the Info. My Pi is running for 2+ days now (never worked so long without a crash).

How did you guys disable it? Just comment out the discovery of it in the config?

EX: discovery: ignore:

(with appropriate spacing of course, github formats out the spacing)

eiiot commented 3 years ago

I would also try disabling the IPP (Internet Printing Protocol), as that also caused a crash for me, although that might have been unrelated.

jherby2k commented 3 years ago

I think you're all chasing distractions. This is clearly a lower level device driver / kernel / boot loader issue. Disabling stuff probably suppresses the bug because the Pi is less busy and disk queues don't fill up. Its not a fix. Wait a few days and you'll see.

eiiot commented 3 years ago

You're probably right, although I've not had any problems with resources running out on my Pi, I'm usually sitting at 10-15% Ram and 0-10% CPU. I don't disagree with you, I'm just saying that this could be caused by an addon, since obviously not every homeassistant user is having this issue

bschatzow commented 3 years ago

I think no one currently understands the issue. Many open github issues with Pi lockups and nothing is close to being resolved. A spreadsheet was add to 1119 issue on the freezing and many users added their configurations. Nothing seems out of the ordinary. Most users are not seeing any memory / CPU issues. Just freezing after several hours to a couple of days. I can run my identical setup on OS 5.4 and it has never failed.

Salamandar commented 3 years ago

@bschatzow Agreed. This issue is disregarded by Home Assistant devs/maintainers, so it's no wonder this issue does not move a bit.

diederik-maes commented 3 years ago

Glad to read I'm not the only one experiencing issues. Since a couple of weeks on my RPi4 with SD regular crashes (every couple of days). Then I finally got it running on an SSD with ZHA back up ad running. Since about 1 week daily crashes. After cold start everything works great. Next morning crashed again. Just moved back from SSD to SD now and hope to have less crashes... All info on getting this fixed would be great. It worked great since half December when I moved to HASS but since a couple of weeks a lot of problems. Maybe the new database? Also see the backups growing fast in size. Disabled MQTT because I don't use it but to no avail for now.

bschatzow commented 3 years ago

Not sure if it makes any sense commenting on this issue as it is closed. Don't think any developers monitor closed issues. There are many other open issues that may be a better place to comment.

diederik-maes commented 3 years ago

Not sure if it makes any sense commenting on this issue as it is closed. Don't think any developers monitor closed issues. There are many other open issues that may be a better place to comment.

Thanks for the comment. Didn't notice it was closed, thanks

GilDev commented 3 years ago

Disabling Google Cast integration helped, but yes not sure it completely fixed it.

BWilky commented 3 years ago

+1 having exact same issues. Pi4, SSD. Crashing every day

GilDev commented 3 years ago

To follow up, indeed disabling Google Cast hasn’t fixed the issue entirely but delays the time before it crashes. I have set up an automation that reboots the system once every day, will see if that helps.