Freeze Everyday

[chris8837]

home-assistant.log

Home Assistant release with the issue:

0.81.0.dev0

Last working Home Assistant release (if known):

It has been happening for a while

Operating environment (Hass.io/Docker/Windows/etc.):

RaspPi 3 Component/platform:

No idea

Description of problem: So my instance of HA has been stopping around 12:30-1:00pm everyday. I can't get the log because I have to hard reset to SSH or Samba in there. I posted this on the HA forums, and they thought it might be related to Zwave. I decided to disable the "heal"function as someone told me it updates around this time. I think it updates at 12:00am though.

For some reason today I was able to check HA at 1:15pm or so and I found my log file to be full of all kinds of errors. I have attached this to the post here. I've never seen my log so full of things. When I look at the logbook in HA Frontend I see that at this time everything becomes unavailable and reconnects in the same 12:33pm time. I cannot figure out what's happening here as quite a few have had HA up for 6 mo. I have configured tasmota with MQTT connection. Homeseer 100+ zwave with the aeotec USB dongle. Opengarage. Also I have Emulated Hub for Alexa, and Homekit configured for Siri.

I dont' have any scripts or automations set for this time. This has been happening for a while and i just would reset it with a tasmota switch when I see that it was down. If I dont' get to it, it can be 3:00 and the instance is doing nothing the entire time.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):

Additional information:

[cgarwood]

From the log you posted, it almost looks like your network dropped out for a minute.. but that shouldn't hard lock the Pi.

Try leaving an SSH connection open running top to show load average and the process list. See if the SSH session gets disconnected or freezes around the same time that HA stops/freezes. If so, see what the load average is and what processes are running at that time.

If that doesn't help, I'd try disabling 1 component per day and see if you can narrow it down to any specific components. I'd probably start with opengarage or the denon media player since they're the first one to show up in the log.

[timothybrown]

II can’t imagine how HA would be knocking the network offline (it doesn’t have the required privileges to mess with the kernel or anything), unless it’s just spawning and endless number of threads that are maxing out processor and I/O time, preventing various services (smbd, sshd, other HA threads, etc.) from answering network requests.

Do you have a monitor hooked to your Pi? Or, better yet, do you have an FTDI or other USB to TTL Serial cable available? (Another Pi, an Arduino, etc. would work too.) If so, I’ll give you some commands that will enable debug level system logging out to the serial port. You can run a Serial terminal on your PC to monitor the log.

This would be much more telling than trying to parse the Systemd Journal after the fact (which might have trouble actually writing the logs if it’s a hardware problem like a bad SD card or something).

[VDRainer]

@timothybrown , im very interested how to enable debug level system logging out to the serial port. Could you post the commands here with some brief explanation? Thanks in advance!

[chris8837]

So my second power supply was bad as well. These pi’s are so finicky I’m seeing.

I built my own and it now seems stable.

[timothybrown]

@timothybrown , im very interested how to enable debug level system logging out to the serial port. Could you post the commands here with some brief explanation? Thanks in advance!

Here’s the straight line approach:

On your Pi, add the following to /boot/cmdline.txt: console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1

This command enables console output on the serial port, plus kernel debugging output.

Note: If you’ve got a Raspberry Pi 3 or above replace ttyAMA0 with ttyS0.

Now, hook your (3.3V) USB to Serial cable up to the appropriate GPIO pins for your Pi (this varies by model, lookup a pinout for yours). Remember to hookup the cable in crossover fashion as so: RPi <-> Serial Adapter TXD - RXD RXD - TXD GND - GND

Then you can open the serial terminal on on your computer and boot up your Pi. (TeraTerm is a free emulator for Windows that works well, though PuTTY or even HyperTerm will do in a pinch. For macOS I highly recommend Serial, which is available on the Mac App Store.)

You should see the entire boot process of the Pi appear on your serial terminal, then a login prompt, just as if you had a monitor attached. Login as your normal user and then start the Systemd Journal in ‘tail’ mode: sudo journalctl -f (If you only want to follow the output of a specific process you can append -u <unit>.service to the above command: sudo journalctl -f -u home-assistant.service, sudo journalctl -f -u mosquito.service, sudo journalctl -f -u postgresql.service and so forth.)

So now you should get kernel debugging and all logs spit out to your serial terminal. Let me know if you need more details on anything.

[timothybrown]

So my second power supply was bad as well. These pi’s are so finicky I’m seeing.

I built my own and it now seems stable.

It’s not that they’re finicky, it’s that most USB supplies are designed for cell phones. Why is this important? Well, a cell phone doesn’t strictly require 5V to charge the battery; most can work down to 4.4V or so (Li-Ion batteries charge at 4.2V). So, while most USB power supplies can deliver their rated current, it won’t be at 5V! Instead it will be at 4.6 to 4.4V.

The issue with this is the Pi really requires a voltage of at least 4.65V to function properly. So, why this discrepancy? Well, the answer is a bit complex, but the gist is a mix of cost savings on the part of the USB power supply manufacturers and the current handling capabilities of the micro USB connector. The pins that transfer power in a micro USB connector are teeny tiny and have a fairly high resistance. Ohm’s Law tells us that over a given resistance voltage will decrease as current increases; the higher the resistance the higher the voltage drop. 3 or 4 amps is the realistic current handling capability of a micro USB connector, and that is pushing it. (As an aside, this is the reason why USB-C uses a much higher voltage at a lower current, which is then transformed into a lower voltage at a higher current inside the device.)

So, you really need a high quality USB cable (one designed for power transfer, ideally with 20AWG wire inside) and a power supply designed for powering and not charging.

What I do is use a micro USB to female barrel jack adapter combined with a high quality non-USB power supply (one with a 5.5mm barrel jack connector on it). Works well and provides more than enough current at all loads.

[VDRainer]

@timothybrown , Thanks very much for this detailed instruction.