search

InfiniTimeOrg / InfiniTime

Firmware for Pinetime smartwatch written in C++ and based on FreeRTOS
GNU General Public License v3.0
2.79k stars 947 forks source link

lv_task_handler can fail to return in high load scenarios, causing watchdog resets #2124

Open mark9064 opened 2 months ago

mark9064 commented 2 months ago

The LVGL task handler re-evaluates higher priority tasks after executing lower priority ones The animation task is higher priority than refresh tasks If the animation task or refresh task takes a long time, after executing the refresh task the animation task may be ready again This causes lv_task_handler to infinitely execute animation->refresh->animation->refresh etc without ever returning If lv_task_handler doesn't return, the DisplayApp message queue becomes full (usually with touchscreen events) and blocks SystemTask, causing watchdog resets

This happens very rarely or even never in standard InfiniTime. But by either decreasing the LV_DISPLAY_REFR_PERIOD and increasing animation load (e.g animation scroll speed for labels in px/s) (I have not tested if both are required to trigger this behaviour, but I know that both are sufficient), it's possible to reproduce this lockup.

I'd suggest that we replace the LVGL task handler with one that provides a hard upper bound on the number of task executions. Perhaps by only executing each task once and exiting early if it's already spent too long doing work (many tasks waiting)? This would allow task priorities to still be usefully enforced. Lower priority refresh tasks might still get starved by animations though.

Possible cause for #2012 (the notification header scrolling animation)

mark9064 commented 2 months ago

Looking at LVGL main/master, they've replaced the scheduler with one that does have bounded execution time (provided timers/tasks are not being created or destroyed). Task priority has also been removed at some point. Both seem like sensible decisions. Removing priority is out of scope, but I think a scheduler that ignores it will be fine. We should really just bite the bullet on upgrading LVGL at some point

mark9064 commented 2 months ago

Even after changing the scheduler to a bounded time one locally I can still trigger lockups. So I think my original theory for the mechanism must be incorrect, there must be some other way for the tasks to take a long time to run. The animation code in LVGL isn't too easy for me to read