Device is freezing constantly after several hours

[marjue12255]

Bug report

Describe the bug

The device is constantly freezing after several hours (5-7). When this happens the display is showing the last action forever. It's not possible to ping the device over the netwok.

Additional information

• Devices involved:
  • Model: Ulanzi Awtrix Smart Pixel Clock 2882 (TC001)
  • AWTRIX-LIGHT version: v0.90

To Reproduce

I have stopped to send my own messages to the device to see if this is the reason. No luck. I have reduced the active app to the clock and date only. No Luck I have disconnected it from the MQTT (Mosquito). No luck. I did a reflash. No Luck

Expected behavior

Should not crash

Logs

I have tried to get logs but they have been empty. Maybe It should be connected while it's crashing?

Additional context

Is there another way to get debugging information to help you (and me). I only know the "Logs and Console" button at he web-flasher.

[Blueforcer]

Seems like a hardware error to me. If this would be a software problem I would have thousands of users with the same problem. Since it also happens with only time and date, I'm pretty sure this would happen to everyone.

do you have a single ESP32 laying around? Flash awtrix on it and check if the Webinterface survive longer than 6 hours

[sockmonkey0223]

In case this helps, I saw this issue for months on my device with different software versions; I had written it off as bad hardware. On a fluke, I changed to powering the device with a proper plug-in USB power supply instead of from a USB hub and the issue hasn't occurred in months.

[lukanvanderlinde]

I'm having the same issue. In my case I tried to reboot and now I perceived a pattern. My crashes everyday around 5-7 in the morning.

First i thought was one of my automations that sends a "water plants" notification in the morning. But after removing the app and disabling the automation, the problem persisted.

Mine is also connected to a gan charger from ugreen alongside some other things. Things that are always plugged so I didn't thought that could be related to a power delivery issue.

I will follow @sockmonkey0223 tip of having a dedicated charger. If fixes the issue I will share here.

[p0se]

I have a similar problem, maybe even the same one?

My device also freezes after 5-7 hours of use. I can still reach it and the live display also works correctly. After a restart, the display remains black. The clock is accessible in the network.

The only thing that helps is to erase and flash the watch again, then the display will work. The error is then repeated the next day after 5-7 hours of use.

I use the firmware 0.90

[lukanvanderlinde]

It's been a week since I did the experiment changing to a exclusive charger and got mixed results.

It keeps freezing but now the network keeps active. Before, Uptimekuma wasn't able to ping, now it does.

Since it last around 24 hours, I will try to automatically restart every night to see if does something.

[p0se]

If my watch is freezed and i let it run for a few days, it will sometimes work again.

Unfortunately, only for a few hours.

In the meantime, it displays shows strange pixels. (See picture). https://imgur.com/a/9X4lkuv

[docchaoswizard]

Yep, i can confirm this is happening on two brand new Ulanzi Clocks. The clock simply crashes and works normally when you turn it on again. I also tried sending a Heartbeat-Reminder every 10 minutes but to no avail. You can see in the smokeping-plot that the clock simply suddendly stops responding to pings.

How can i help to fix this issue? Its sadly really frustrating...

For reference: I am using Awtrix 0.91. The crashing occurs while running via USB and battery (But the battery is not empty)

[abl73]

Not sure my issues are the same as these, but with the firmware 0.91, my esp32 device hangs and then restarts regularly. Rolled back 0.87 firmware and now my Awtrix is running perfectly again...

[docchaoswizard]

Not sure my issues are the same as these, but with the firmware 0.91, my esp32 device hangs and then restarts regularly. Rolled back 0.87 firmware and now my Awtrix is running perfectly again...

Good idea. Thanks. I will try that and report back. Did you randomly try 0.87 or did you work your way backwards?

[Blueforcer]

I think downgrading doesn't help to find the error. Does it freeze directly after flashing (with erase)? What does the USB console output say at crash?

[Blueforcer]

Yep, i can confirm this is happening on two brand new Ulanzi Clocks. The clock simply crashes and works normally when you turn it on again.

Does it crash and reboot, or turn off by itself and you need to starts it with the two buttons again?

[docchaoswizard]

Yep, i can confirm this is happening on two brand new Ulanzi Clocks. The clock simply crashes and works normally when you turn it on again.

Does it crash and reboot, or turn off by itself and you need to starts it with the two buttons again?

Yes, it turns off by itself (And won't respond to pings or MQTT, obviously) and I have to turn it on again, after that it boots normally and runs for another few hours.

Downgrading to 0.87 also didn't fix the issue

Yes it also froze directly after I flashed it with the web-installer. I will try erasing the flash and flashing again with esptool and report back. I'll think about how I can record the serial output over several hours.

Cheers

[Blueforcer]

Im confused. Freeze and turning off is not the same.

If awtrix went off ( black screen) etc. And you need to turn it on by holding both buttons, It's 100% not a software problem, because it's not possible to turn the hardware off or on by software. There is simply no connection between. Use a more powerful power supply and don't use the cable wich comes with the clock. It sounds like your battery discharge over time, if it's too low the ulanzi turns off (even it's plugged in)

[docchaoswizard]

Im confused. Freeze and turning off is not the same.

If awtrix went off ( black screen) etc. And you need to turn it on by holding both buttons, It's 100% not a software problem, because it's not possible to turn the hardware off or on by software. There is simply no connection between. Use a more powerful power supply and don't use the cable wich comes with the clock. It sounds like your battery discharge over time, if it's too low the ulanzi turns off (even it's plugged in)

A 5V/3A Charger with a good quality cable should be sufficient, or? I checked, the clock just turned off with 72% battery so i don't really think this is the issue.

When you send {"power": false} to the /power Endpoint of the clock it turns off the matrix, as expected. To turn it back on you could either send {"power": true} or hold both buttons, after a while the clock beeps and boots again. So i am guessing something like this is happening. Something causes the matrix to turn off and disable the wifi. Holding both buttons causes it to reset after a a few seconds.

Just to make sure nothing is interfering with the testing i also used a different MQTT-Prefix to make sure no Home-Automation is doing anything but to no avail. The clock ran for just over an hour directly after flashing and wiping and for about 3 hours after resetting. I am currently recording every serial message, maybe we will catch something.

Cheers

[Blueforcer]

Power Command only turns the matrix off to 0% Brightness. Software is still fully running.

If you only need to press both buttons once to power it on again, it's completely off. If you hit both buttons twice (turn off and then turn on) then it's a software part.

[luismalves]

I'm also having this issue from some hours to hours, device is new. Anything I can get to help the debugging process?

[Blueforcer]

I'm also having this issue from some hours to hours, device is new. Anything I can get to help the debugging process?

What issue? It seems that problem here is, that awtrix turns off itself. The solution is to use a better power supply because the battery simply discharges faster than you charge it.

[captorg]

I have a similar issue: display is freezing after a while - even when a turnover from calender app to costumized app is performed, device is not reachable afterwards and needs to be restarted manually.

Conditions: Device powered via USB, firmware v0.94

[luismalves]

I'm also having this issue from some hours to hours, device is new. Anything I can get to help the debugging process?

What issue? It seems that problem here is, that awtrix turns off itself. The solution is to use a better power supply because the battery simply discharges faster than you charge it.

Exactly, thank you! That was the issue and might help others: with full brightness, some power supplies dont provide enought power so it would crash it. I've solved for now with Home Assistant turn it off at 40%

[captorg]

I'm also having this issue from some hours to hours, device is new. Anything I can get to help the debugging process?

What issue? It seems that problem here is, that awtrix turns off itself. The solution is to use a better power supply because the battery simply discharges faster than you charge it.

Exactly, thank you! That was the issue and might help others: with full brightness, some power supplies dont provide enought power so it would crash it. I've solved for now with Home Assistant turn it off at 40%

My clock runs at 20% brightness, is powered by a 2A power supply and still crashes...

[marjue12255]

Hello I haven't been in touch for a while after I opened this bug. But I have followed it and followed the tips regarding the power supply. But it didn't help. The problem persisted. I then ordered another Ulanzi and solved the problem with hardware. I have now reactivated the old "defective" Ulanzi and updated it to from 0.91 to 0.94. As I said, an update. The settings have remained the same. The watch has now been running for a week without any problems. Strange.

Greetings Marcus

[Bazz-Man]

Ive opened a bug report #485 which I think is easily reproducable and could be well why people are seeing the issues above and also why they are intermitant and/or seem to disappear ?

[Jeroenimootje]

It is odd that mine is going to AP mode after several hours, but I am still able to browse the GUI, so it is still connected to the network. The live view tab is saying "file not found", but I am still able to restart the ESP using the "network" or "icon" tab in the GUI. The battery doesn't go below 90% in my setup. I am using it in Home Assistant, but unfortunately I was not able to find a way to automate the restart as a work-around.

[Jeroenimootje]

It is odd that mine is going to AP mode after several hours, but I am still able to browse the GUI, so it is still connected to the network. The live view tab is saying "file not found", but I am still able to restart the ESP using the "network" or "icon" tab in the GUI. The battery doesn't go below 90% in my setup. I am using it in Home Assistant, but unfortunately I was not able to find a way to automate the restart as a work-around.

I found the resolution. Just give it a fixed IP address instead of using DHCP.

[Blueforcer]

Hello I haven't been in touch for a while after I opened this bug. But I have followed it and followed the tips regarding the power supply. But it didn't help. The problem persisted. I then ordered another Ulanzi and solved the problem with hardware. I have now reactivated the old "defective" Ulanzi and updated it to from 0.91 to 0.94. As I said, an update. The settings have remained the same. The watch has now been running for a week without any problems. Strange.

Greetings Marcus

The author's main problem is solved. Please open another issue if you have different problems