InfiniTimeOrg / InfiniTime

Firmware for Pinetime smartwatch written in C++ and based on FreeRTOS
GNU General Public License v3.0
2.64k stars 904 forks source link

Out of memory using weather with Gadgetbridge, causing font corruption #1788

Closed Hegz closed 5 months ago

Hegz commented 1 year ago

Verification

What happened?

Clock text becomes corrupt

What should happen instead?

I was expecting to see readable fonts

Reproduction steps

I've updated to 1.13, and enabled weather with gadget bridge and quickweather. After about 9 hours being on on my wrist, I glanced at my watch and the clock font's were corrupt. This has been happening for the past several days after updating to 1.13 and enabling weather support. I don't know the time periods for the other times.

The following are corrupted:

Unaffected faces

More details?

I can switch to an unaffected face, and the watch will continue to work as normal. Attempting to flip back to Pinetimestyle causes the screen to lock before displaying the Pinetimestyle face, and then it reboots.

The previous reboot was at about 6:00am, which was a spontaneous reboot. I didn't see if there was corruption prior to the reboot. The corruption kicked in at around 3:00pm the same day.

I've attempted to reinstall the assets pack, as I thought it may have changed since I first installed it. This didn't change the behaviour.

Version

1.13

Companion app

Gadget Bridge

Hegz commented 1 year ago

The text corruption appears the same as shown on #1376

FintasticMan commented 1 year ago

When this happens, can you check the values shown under "Memory heap" on the 3rd page of the About page (you can get there quickly by holding the button for 2-3 seconds)? I have a feeling that that is caused by running out of memory.

kieranc commented 1 year ago

The text corruption appears the same as shown on #1376

I disagree, this is a new and different corruption, sadly. It's now more transparent, with vertical banding. The extra strange bit is that PTS fonts are internal, previously only fonts loaded from ext flash were affected but now it seems to be affecting internal fonts as well. I saw this behaviour a couple of days ago but it only affected the temperature label on PTS, while I was provoking it, so I thought I was to blame... but apparently not.

JF002 commented 1 year ago

Yep, I agree with @kieranc, I don't think this issue has the same cause than #1376 since PTS does not load anything from the external storage.

But it might still be memory related : the weather service does use dynamic memory allocation. If the memory is not correctly released, or if the weather service receives too many notifications, the memory could be too fragmented, or completely filled, which would prevent the watchface from loading correctly... Well this is just an hypothesis since I've never encountered this issue on my PineTime.

@Hegz would you like to post one or two of pictures of the display when the text is corrupted?

Also, would you like to disable the weather functionality from your companion app and see if this issue still occurs? I would like to know if this issue is linked to the weather service or to something else.

And @FintasticMan 's suggestion is also interesting:

When this happens, can you check the values shown under "Memory heap" on the 3rd page of the About page (you can get there quickly by holding the button for 2-3 seconds)? I have a feeling that that is caused by running out of memory.

hubortje commented 1 year ago

I experience a similar issue, yet my runtime is way longer than 9 hours. I'm not quite sure, but my watch is running since the 1.13.0 update was released.

The corruption is cleared by swiping into one of the adjacent menus, and swiping back to the watchface. Pressing the power button repeatedly to turn the watchface off and on again, does not solve the issue. Did not try to reboot the device tho.

Find a picture of the corrupted clock attached.

Corrupted watchface!

Sokoloft commented 1 year ago

I am also having the same issue. Below are a couple pictures of both the PTS watch face and Casio watch face being affected.

IMG_20230702_210042453_HDR

IMG_20230625_204016905_HDR (1)

I will try to look at the memory information as @FintasticMan suggested. However the watch likes to crash whenever this happens.

JF002 commented 1 year ago

I will try to look at the memory information as @FintasticMan suggested. However the watch likes to crash whenever this happens.

Yeah, this is likely to happen in case of memory allocation error, unfortunately. Anyway, you could still have a look at that screen from time to time, and especially have a look at the "min free" value (which shows the lowest amount of memory available since InfiniTime was started on your watch) and "alloc err" (allocation error) and "overfl err" (overflow error). Ideally, these two last value should always be 0.

hubortje commented 1 year ago

random shower-hypothesis: I had no trouble regarding the corrupted clockface, until I switched the settings in TWFG from a six-hours-interval, to an hourly-interval – maybe this has to do with the issue?

jwest23 commented 1 year ago

Possibly related, I've seen a couple of random reboots of my watch. I don't know if it's happening when I'm not looking, too. I have physically seen two reboots, though

Sokoloft commented 1 year ago

So far I'm 8h 31m uptime and min free value dropped a tiny bit. It started at 4744 and now its at 4736. Idk if that helps any. I'll monitor it closely tomorrow and see if I can get more uptime where I can monitor it.

Hegz commented 1 year ago

PXL_20230704_022437529 PXL_20230704_022431169 PXL_20230704_022424973 PXL_20230704_022410995

I don't recall seeing the corrupt face since I reported this, but I also didn't reboot this at 4:30.

Here are images of the corrupt watch face, as well as the Diagnostic menus that I was able to access. I attempted to change to the Infinineat watch face, and it rebooted.

I'll attempt turning off weather on the watch-face & companion app (Gadgetbridge, quickweather) and report back in 24h, or when the device crashes again.

Sokoloft commented 1 year ago

With one day and 5h uptime. The min free value dropped to 4560. Worth mentioning I have my weather app not sending data to GadgetBridge. It seems more stable since doing that.

I also deleted the fonts and images folder on the watch and re-install the resource file. However @FintasticMan mentioned that shouldn't make a difference. Its likely the fact I have no weather data going to GB.

Hegz commented 1 year ago

Almost 24 hours without weather data, and I'm not having any stability problems. "Min free" hasn't changed from 8080 all day. Some different values in the 4th screen of About. Not sure if it matters if it dose I already took a picture and can post.

I'll try turning on the quickweather gadget bridge Integration but leaving off the watch face display.

Hegz commented 1 year ago

With the Integration on, and watch display off, it still crashed. it took 27h this time.

Hegz commented 12 months ago

Last update to this, tried watch display weather off, quickweather on, and using the Casio watch face. Crashed at 7 hours.

Memory heap Free 12560 Min free 32 Alloc err 275 Ovrfl err 0

Sokoloft commented 12 months ago

Just to add to the information. My watch has 3d 8h uptime still with 4560 min free. Using the Casio watch face and with the weather turned off in my weather application (fork of forecastie).

So anyone observing this can sum up that something with the weather integration is the issue. However even prior to this release I have had the watch face corrupt in the past. Opening an application on the watch and going back to the watch face would fix it. I think that is a known issue though.

kieranc commented 12 months ago

So anyone observing this can sum up that something with the weather integration is the issue. However even prior to this release I have had the watch face corrupt in the past. Opening an application on the watch and going back to the watch face would fix it. I think that is a known issue though.

Agreed, the previous corruption was different, and seemed to be resolved by the heap unification, this one is new and probably caused by weather data being stored on the watch. I think that removing the 24h forecast from the data GB sends will help but I don't know how much, but more than that, I think we need to set valid expiry times for events on the timeline so they get cleaned up as they expire - currently, even ignoring the 24h forecast, if GB sends data every hour, the events have a fixed 6h expiry time so we could potentially have 5 expired but not cleaned up events in the timeline, even more if we count the 24h forecast. I don't know how much memory this consumes but evidence suggests it's enough to be a problem.

edit: I just prepared a PR for gadgetbridge to remove the 24h forecast and I'm wondering if it would be better to reduce the weather data sent even further to just what we need/use right now? We're sending wind speed and humidity and possibly a condition code, none of which are used.

FintasticMan commented 12 months ago

What I've personally noticed, is that the amount of used memory stays pretty consistent, and then it suddenly gets used up. I haven't had time to test it with a debugger connected, but it seems to me that it might not be entirely due to just there being too many events on the timeline.

Hegz commented 12 months ago

There was a GadgetBridge update with the note "InfiniTime: Fix weather expiry time"

It's still happening after that update. It may have taken a little longer, but was still less then 24h to crash.

segaretroboy commented 11 months ago

Had this happen to me as well, rebooted the watch before I thought to get debug info, but in my instance the weather and other icons on the pinetime style got corrupted too. IMG_20230716_215128

hectorzg commented 11 months ago

I have experienced the same watch face corruption but in addition to that my pinetime screen only showed a black screen and there was nothing I could do to fix it, sometimes the issue fixed itself after a few hours and others I had to reinstall the latest firmware via gadgetbridge. This issue happened with any watchface.

I have downgraded to firmware 1.12 and haven't had any issues in the last couple of days.

zonorti commented 11 months ago

I have experienced the same behavior a few hours after enabling weather push (via Gadgetbridge). Without weather I had stable 10 days of uptime. image

segaretroboy commented 11 months ago

Wanting to add as this has continued on my device, after setting up Quick Weather and turning on Gadgetbridge integration, the watch has begun doing this much more frequently. Confirmed I had Allocation errors each time it happened.

StevenAlexander44 commented 10 months ago

If you use the debug menu in gadgetbridge and spam click "Set weather", then you can see the bug in about 10-20 clicks from a fresh restart, and after about 20-30, the watch completely dies and reboots. After a fresh reboot, for me, at first it shows a blocked symbol "🛇", then after clicking "Set weather" once, it shows the correct weather, then after clicking "Set weather" a second time and after, it starts to show correct temperature shortly, and immediately rewrite with the incorrect number.

Versions of software in testing are: InfiniTime: 1.13 with infineat and casio installed Gadgetbridge: Nightly 0.75.1-20d0d61ed breezy-weather: v4.6.4-beta

vkareh commented 5 months ago

I have not experienced this since the SimpleWeatherService became available in 1.14

AgentConDier commented 5 months ago

Agreed, 13d uptime right now, with weather enabled and working :+1:

FintasticMan commented 5 months ago

Oh yeah, this issue only exists with the old weather implementation.