InfiniTimeOrg / InfiniTime

Firmware for Pinetime smartwatch written in C++ and based on FreeRTOS
GNU General Public License v3.0
2.64k stars 903 forks source link

Always on display #1869

Open KaffeinatedKat opened 9 months ago

KaffeinatedKat commented 9 months ago

I have added an option in the "Display timeout" for an always on display. Not sure how useful this is, or how good the battery life is (will be testing this and will post a comment with how long the battery lasted).

Any suggestions on how I could possibly increase the battery with the display always on would be great.

image

github-actions[bot] commented 9 months ago
Build size and comparison to main: Section Size Difference
text 378708B 1500B
data 948B 8B
bss 63504B -52B
JF002 commented 9 months ago

This will probably reduce the battery life to just a few hours. The LCD uses between 10 and 30mA when it's ON, which is quite high compared to the current power usage of InfiniTime (less than 1mA).

I can't think of any way to reduce power usage of the display when it's on... Lowering the brightness will effectively reduce the power usage, but probably not enough to maintain a reasonable battery life. This is caused by the LCD technology of the display.

KaffeinatedKat commented 9 months ago

is there anyway to reduce the brightness lower than the lowest setting right now? Currently on the lowest brightness setting it's gone ~15 hours and it's at ~35%, which was better than I was expecting

JF002 commented 9 months ago

You could drive the brightness using PWM. There might already be a PR for that.

mark9064 commented 9 months ago

You could drive the brightness using PWM. There might already be a PR for that.

Indeed there is, see #575

KaffeinatedKat commented 9 months ago

I've changed it so that the "always on" setting prevents the display from turning off, but the watch still goes into the sleep mode, and just sets it to a low brightness with pwm. Now the display will light up when any of the wakeup actions happen, then go dim again after the display timeout.

My only issue is the watch is in sleep mode and the watchface does not update while sleeping. Because the display is still on the watchface is just frozen till woken up. I've been poking around the code and I cannot find where I can make the display update while sleeping

A video of the watchface not updating while asleep: https://drive.google.com/file/d/1pNt-cbVS1s_8tmAK6B8ZmDR_WU164OI8/view

mark9064 commented 9 months ago

A common pattern you'll spot all across InfiniTime is using queue read timeout to implement periodic tasks. Many tasks have an event queue that they read from forever, and when fetching an event from a queue you can specify a timeout, which is how long to wait if nothing arrives. After this timeout, the queue fetch function returns that no items were fetched from the queue as nothing was available in the time specified. So if you have a loop which does:

  1. get event with timeout 1 minute
  2. process event if one exists
  3. execute x task

then x task runs at least every minute.

This is exactly how it's implemented for the display, with x task in this case being refreshing display contents. So change

    case States::Idle:
      queueTimeout = portMAX_DELAY;
      break;

to

    case States::Idle:
      if (settingsController.GetAlwaysOnDisplay()) {
        queueTimeout = lv_task_handler(); // returns time until LVGL tasks need running
      } else {
        queueTimeout = portMAX_DELAY;
      }
      break;

and you're good :)

mark9064 commented 9 months ago

Seems to work well with the above change. I haven't been running it long enough to test battery yet though. One suggestion would be to disable always on if InfiniTime is set to sleep mode

KaffeinatedKat commented 9 months ago

Seems to work well with the above change. I haven't been running it long enough to test battery yet though. One suggestion would be to disable always on if InfiniTime is set to sleep mode

This is something I plan to implement, this feature would pair really nicely with #1461

KaffeinatedKat commented 9 months ago

A common pattern you'll spot all across InfiniTime is using queue read timeout to implement periodic tasks. Many tasks have an event queue that they read from forever, and when fetching an event from a queue you can specify a timeout, which is how long to wait if nothing arrives. After this timeout, the queue fetch function returns that no items were fetched from the queue as nothing was available in the time specified. So if you have a loop which does:

1. get event with timeout 1 minute

2. process event if one exists

3. execute x task

then x task runs at least every minute.

This is exactly how it's implemented for the display, with x task in this case being refreshing display contents. So change

    case States::Idle:
      queueTimeout = portMAX_DELAY;
      break;

to

    case States::Idle:
      if (settingsController.GetAlwaysOnDisplay()) {
        queueTimeout = lv_task_handler(); // returns time until LVGL tasks need running
      } else {
        queueTimeout = portMAX_DELAY;
      }
      break;

and you're good :)

I have applied this, and it doesn't work. Screen still doesn't update while sleeping and when I wake it up the colors invert, which is extremely weird. Any idea why?

KaffeinatedKat commented 9 months ago

I have applied this, and it doesn't work. Screen still doesn't update while sleeping and when I wake it up the colors invert, which is extremely weird. Any idea why?

I have solved this issue, this change worked fine in the simulator but did not work on the watch. Figured out that by not putting the SPI to sleep, these changes work as intended

mark9064 commented 9 months ago

Nice job, I'm running a patchset that disables SPI sleep and I suspected that might be difference - unfortunately I didn't have time to test and report back so sorry about that

One change I'd suggest: running the LVGL tasks full speed (ie max display refresh) is pretty power hungry, and I've found throttling it to 250ms saves power while keeping watchfaces with seconds up to date (as the display effectively refreshes at 4Hz). I'm getting ~30h battery life running the screen on constantly right now (This is a calculated number as I turn it off at night) I also thought for correctness it might be good to pull up the screen running check. It also might be good not to reinitialise the SPI inside GoToRunning if it wasn't actually suspended

So I've been running this - feel free to integrate changes if you fancy

      if (settingsController.GetAlwaysOnDisplay()) {
        if (!currentScreen->IsRunning()) {
          LoadPreviousScreen();
        }
        int lvglWaitTime = lv_task_handler();
        // while in always on mode, throttle LVGL events to 4Hz
        queueTimeout = std::max(lvglWaitTime, 250);
      } else {
        queueTimeout = portMAX_DELAY;
      }
      break;

I've also noticed an issue where sometimes the brightness is on Low rather than Lowest while the watch is asleep - I haven't had time to debug that either but see if you spot it. For reference I have my normal brightness set to Low

escoand commented 9 months ago

Wouldn't be an update every minute enough for an always on display? It should increase the battery runtime further.

Off course with hiding the seconds hand...

mark9064 commented 9 months ago

Wouldn't be an update every minute enough for an always on display? It should increase the battery runtime further.

Off course with hiding the seconds hand...

Yeah it's definitely something worth thinking about. On one hand it saves power, but on the other, it allows incorrect information to be onscreen for more than a minute like seconds on watchfaces with them, seconds on the timer app, bluetooth status, music status, steps, heart rate (if continuous/background measuring implemented) etc Whether these are important enough for the power saving, I'm not sure. Maybe it's worth testing to see how big the power difference is?

KaffeinatedKat commented 9 months ago

Wouldn't be an update every minute enough for an always on display? It should increase the battery runtime further. Off course with hiding the seconds hand...

Yeah it's definitely something worth thinking about. On one hand it saves power, but on the other, it allows incorrect information to be onscreen for more than a minute like seconds on watchfaces with them, seconds on the timer app, bluetooth status, music status, steps, heart rate (if continuous/background measuring implemented) etc Whether these are important enough for the power saving, I'm not sure. Maybe it's worth testing to see how big the power difference is?

This might be worth it, we can set the update timer to 4Hz when apps like the timer and music are running, but set it back when not running these apps; perhaps an enum with apps that need a higher refresh timer. We can also disable the seconds on every watchface that has them when always on mode. We could also add a setting to disable this if someone wants seconds on. I think with the display the pinetime has, any power savings we can get is worth it

hexisXz commented 8 months ago

This will probably reduce the battery life to just a few hours. The LCD uses between 10 and 30mA when it's ON, which is quite high compared to the current power usage of InfiniTime (less than 1mA).

I can't think of any way to reduce power usage of the display when it's on... Lowering the brightness will effectively reduce the power usage, but probably not enough to maintain a reasonable battery life. This is caused by the LCD technology of the display.

I honestly think this would be pretty useful. You could just have a low quality version of the digital clock display on the screen with like medium brightness and just have everything else suspended in the background.

KaffeinatedKat commented 8 months ago

This pr, for the most part, is complete. Only remaining issue is sometimes the display does not go into the lowest setting when going to sleep, and stays at the brightness setting. I have not been able to figure this out because it happens so infrequently.

@JF002, is there any way this feature would be considered for merging, or is the battery life reduction while enabled to high? Maybe there could be some kind of popup warning when enabled, to notify the user of the significant battery drain while enabled.

It lasts about a full day when enabled, and most smartwatches only last that long anyways. I feel like the people who genuinely find this useful would be willing to charge it everyday

mark9064 commented 8 months ago

The issues with the screen getting stuck in Low are due to the restorebrightness messages coming from the system task. (Edit: I can post some actual patches soon)

I think this PR (with the above issue fixed) is usable. But we should also test what the majority of the power consumption is when always on (I suspect it's the LCD) and if we can reduce it further (the LCD offers a power mode between sleep (display blank) and fully on with reduced colours etc.). Also might be possible to redrive the LCD at a lower refresh rate when in always on, but the datasheet for it on the Pine wiki is an absolute monster and I haven't had the time to fully explore it. As an aside it looks like the solution to the display invert issue might be in there too

KaffeinatedKat commented 8 months ago

I took a peek at the datasheet, and implementing switching to reduced colors while always on. The datasheet says this uses less power, but by how much I am unsure. I don't have a devkit, and can't measure power consumption.

This is also essentially free power reduction. Turns out, InfiniTime doesn't utility many colors at all, and while in reduced colors, you can hardly tell. The reduced brightness also helps hide it. The one app where you can really tell is 2048, and I doubt anybody will be letting it fall asleep on 2048 anyways.

I'm gunna keep poking around the datasheet to see if I can figure out the refresh rate thing, if that is possible it would probably help the power consumption by a sizeable margin

mark9064 commented 8 months ago

Oh yeah also forgot to mention, the settings version number should be bumped as a new member has been added

mark9064 commented 8 months ago

https://github.com/InfiniTimeOrg/InfiniTime/pull/1869#issuecomment-1762923811

The patch in question: https://github.com/mark9064/InfiniTime/commit/d43759fcc817be8b1daaadc7bd02f5353be06e47 Feel free to pull the commit into your branch if you're happy with it :)

Locally working well - to reproduce the issue on the original try disconnecting and reconnecting BT (the display should get stuck on after). With the patch it should stay in always on as expected

Reduced colours mode seems to be working, only just flashed it so haven't had time to see if there are power changes (I only have sealed). Some of the watchfaces suffer though

KaffeinatedKat commented 8 months ago

Applied your patch, and it has fixed the issue on my device as well. Bumped the settings version, while I was at it

Also figured out how to modify the refresh rate, it now goes to ~4.8hz while always on, which is the lowest setting available. Hopefully this helps increase the battery life

If anyone with a devkit wants to test if the reduced colors actually saves a realistic amount of power that would be great. I feel like it would be worth reverting if it doesn't save enough power due to making some watchfaces look worse

mark9064 commented 8 months ago

+1 on the reduced colours thought

Can confirm refresh rate changes have applied properly, well done on figuring that out! I think it should actually be possible to inform LVGL of the exact refresh rate too and that way we shouldn't have to do the timer throttling hack? I'm not totally sure on this, I'd need to dig through the LVGL internals but it might be worth considering

mark9064 commented 8 months ago

I had a quick look: it seems that LV_DISP_DEF_REFR_PERIOD is used as a define to set the refresh period on all the LVGL tasks to 20ms. So I think what we'd have to do is go through all of these tasks (there are many inside InfiniTime and a couple in LVGL itself), and then use lv_task_set_period to set them to a longer execution period. Quite a bit of work, but looks possible

Edit: Damn this is looking pretty complicated. Some of the tasks should be suspended altogether (like the input reading task) while some should just be slowed. Not sure how feasible doing this is

Edit 2: The more I think about it, the less this seems like a good idea. Trying to bring all of the LGVL tasks together to change the intervals sounds like a bit of a nightmare organisation wise. Perhaps it would be better to properly understand exactly what lv_task_handler does and see if we should implement our own task handler that runs tasks with efficiency in mind rather than latency (ie batching task execution). What are your thoughts?

KaffeinatedKat commented 8 months ago

Perhaps it would be better to properly understand exactly what lv_task_handler does and see if we should implement our own task handler that runs tasks with efficiency in mind rather than latency (ie batching task execution). What are your thoughts?

I think that this is a good idea, would this new task handler replace lv_task_handler all together, or only be used for the always on display?

mark9064 commented 8 months ago

Probably just for always on as I suppose we still want responsive inputs etc while on. But I guess it depends on how it's implemented? It might even not be needed at all - I'm not sure how the LVGL scheduler works

A quick list of things I've noticed so far:

Battery life is looking promising though! Not sure if it's the reduced refresh or reduced colours but I'm seeing an improvement for sure

KaffeinatedKat commented 8 months ago

Screen tearing is quite visible

I've noticed the screen tearing, it's really bad if you run it at that refresh rate while awake. I don't personally run a watchface with seconds and so I haven't noticed it much, but it's for sure worth investigating for those who do. I'll do some tinkering with the refresh rate and see about VSYNC

A background transition to the Sleep mode (eg with the auto sleep PR) causes everything to break

I have encountered this, stopping the spi from sleeping altogether fixes the crashing, and then I just call "GoToRunning" to fix the screen state being wrong afterwards (see e6d60f1). We should find a way to fix this bug without the autosleep pr however, i'll keep you updated on that

There are still some wake/sleep state bugs I need to investigate (display gets stuck on in some scenarios)

I have not encountered any of these, if you could provide some scenarios that cause this, I can help debug it

mark9064 commented 8 months ago

Coming in on 48h runtime now with 20% still in the tank, very impressive! I'd say it's a lot more usable with those changes, awesome job

Another thought I had for power usage is driving the LCD at a slightly lower voltage. From what I can see the LCD driver allows choosing what drive voltage is used for the LCD, so maybe a slightly lower voltage could save some power? It also might cause the LCD to not turn pixels on/off properly so not sure if this is a viable route (and without a power profiler I'm not even sure if the panel draws much power or if it's mostly the driver chip). Also there appears to be about 10 driver voltages :grimacing: so yeah sounds fun

One other thing: some of the comments from the original PWM PR should probably be addressed. I think the points about not running the PWM engine when the screen is off or at one of the fixed brightness modes has merit, with PWM only being used between brightness settings. I think it also makes sense to PWM just one pin at a time, though I can see the formulas for calculating brightness being annoying as the brightness increase per duty increase is different for each pin Ideally I think this should take the form of a unified brightness interface where it takes some target brightness value between 0-10000 (or whatever) and the brightness code calculates what pin configuration is needed along with whether PWM is required? I could have a crack at implementing this, what do you reckon?

KaffeinatedKat commented 8 months ago

I can go ahead and mess with the voltage stuff, still messing around with the refresh stuff to reduce screen tearing, might as well mess with the voltage while i'm at it. Really glad to see how much potential there is for greater battery life with the display being on all the time, considering it's just a regular LCD screen

About the PWM stuff, I honestly have no idea how it works, but if you wanna take a crack at implementing the unified brightness interface, I would happy merge that into the branch. Sounds like it would be a really nice addition to improving the PWM implementation.

KaffeinatedKat commented 8 months ago

I have changed all the voltage settings to their lowest and have been running it for a while and everything seems fine, the display still updates normally and I can't notice any pixels failing to turn on/off. Best case scenario, we save a bit of power, worst case, it does nothing. Really wish I had a devkit and power analysis tools to see if this makes any difference

KaffeinatedKat commented 8 months ago

Frame pacing is noticeably poor

This has really been bothering me. The delay that the DisplayApp runs at is not consistent, and increasing the delay it runs at creates really inconsistent draw times. The longer you make the delay, the worse it gets.

Normally, the DisplayApp updates so often that this is a non-issue for watchfaces with seconds. It grabs the current time so fast and redraws the screen every time, so it's always up to date. When you increase this delay, it only grabs the current time every so often, and because the timing is inconsistent, this leads to the seconds skipping, or moving between seconds quickly every so often.

I don't know why this is, and i'm wondering if there is some kind of workaround to make the DisplayApp update at a very consistent rate

mark9064 commented 8 months ago

I've poked around too, and in fact I'm also pretty sure it's due to the desynchronisation between displayapp and the screen refresh Adaptive sync on desktop PCs works by refreshing at full scanout speed and then waiting for the next refresh cycle. At maximum refresh rate, the wait is zero, and it becomes longer as the input refresh rate decreases below the monitor refresh rate I have discovered that the pinetime LCD controller does not work like this. The scanout slows as the refresh rate drops. I find the scanout to be quite visible in the 2s app when in always on mode, look for a sweeping bar going top to bottom. If you can't spot it try a darker room

I think this can be resolved by synchronising the LVGL refresh rate with the display refresh. Ideally, we'd using VSYNC. I think that may be impossible as the pin for VSYNC is not connected as far as I know, and also it disables features like hardware scrolling. Instead the displayApp loop must be changed to be a time-locked loop that runs lv_task_handler exactly at at the display refresh interval. Time locked loops are kinda tricky (I've implemented one for the heart rate sensor as part of some work I've done) as you have to deal with things like system tick count wraparound, being CPU starved, and avoiding overflow and imprecision when calculating the required sleep time. I can definitely take a look at implementing this.

This still isn't a clean win as there will be a tear line at the same place every refresh. But the pacing should be better?

mark9064 commented 8 months ago

Oh yeah one other issue I've noticed, after leaving the flashlight app brightness is not restored. Quick todo

KaffeinatedKat commented 8 months ago

The patch for InfiniSim is really easy, it won't compile because it implements it's own BrightnessController which doesn't have the Lowest setting in the enum. I'll go ahead and open a PR for that (InfiniTimeOrg/InfiniSim#124). I'll allso check out the other voltage stuff

Another issue i've found is sometimes the LCD turns off after waking up (data is sent to it out of order causing it to shut off, a reboot is required to undo this), which stems from the refresh rate changing. I am pretty sure this is because the delay after setting it is not quite right. I'm currently testing out different delay times to get rid of this issue.

I can also go ahead and check out the flashlight bug you mentioned

KaffeinatedKat commented 8 months ago

Another issue i've found is sometimes the LCD turns off after waking up

After a day of testing, i've figured out when this issue occurs. If you are running the analog watch face with the always on display, sometimes when you wake up/put to sleep the watch, the screen blanks out till you reboot. This only happens on the analog watch face. Every other watch face works, all the apps that allow sleeping also work. As for why this issue occurs, I have no idea.

mark9064 commented 8 months ago

Huh, interesting. I'll see if I can figure out why

Edit: Reproduced. FWIW I haven't applied the flashlight brightness fix yet. Looks like the backlight is coming on fine, but nothing from the LCD

KaffeinatedKat commented 8 months ago

Looks like the backlight is coming on fine, but nothing from the LCD

Yea, calling SoftwareReset doesn't bring back the LCD after this happens either, which is quite strange. I'm going to attempt to figure out what is happening if you wanna focus on the other stuff instead, you have a bit on your plate already

KaffeinatedKat commented 8 months ago

If I remove the second hand on the analog watch face, the LCD going blank stops happening, ran it for a day and I never encountered it. Went ahead and implemented a simple check that disables the seconds hand while always on as a fix. Ideally, we find out why this issue is happening in the first place and fix that, but in the meantime we can prevent the issue from appearing

mark9064 commented 8 months ago

No exciting progress yet but

mark9064 commented 8 months ago

If I remove the second hand on the analog watch face, the LCD going blank stops happening, ran it for a day and I never encountered it

Super weird, maybe there is some implementation issue with the analogue face? Other watchfaces seem to work fine. Could also be that we are violating some assumptions somewhere in the watchface code or LVGL. I doubt it's the display as it gets treated the same no matter the watchface - unless it's something to do with the render time of the watchface

mark9064 commented 8 months ago

Implemented gate control, you should just be able to hit merge on https://github.com/KaffeinatedKat/InfiniTime/pull/1

KaffeinatedKat commented 8 months ago

unless it's something to do with the render time of the watchface

This is my best guess. It calculates Sin and Cos (these are notoriously slow), and also it draws the hands with lines defined by 2 points. Depending on what kind of line drawing algorithm LVGL uses, this could also be quite slow. It does these every second. It could be that because of the time it takes to draw, it sends the image to the display at an inopportune time while the refresh rate is changing causing the display to freak out (overflow? race condition?). If this was the case, it would explain the inconsistency, and also why removing the calculation and drawing of the second hand prevents the issue

mark9064 commented 8 months ago

So I took a look at the fix for flashlight brightness, and I thought it could be simplified by making it flashlights job to restore the brightness. I think this works better as forcing the brightness to be set on every button press is restrictive to functionality and creates weirdness for when the flashlight app is left without the button being pressed (eg by notification).

Does the PR workflow work for you btw, it's definitely more convenient for me than copy pasting patches and keeps the blame accurate so I can be called out when everything catches fire https://github.com/KaffeinatedKat/InfiniTime/pull/2

KaffeinatedKat commented 8 months ago

and I thought it could be simplified by making it flashlights job to restore the brightness

This is much more elegant, also fixes an issue where if you wake up the watch and put it to sleep right away with the button it would go to sleep then set the brightness to normal.

Does the PR workflow work for you btw

Yes, much better than before. Much easier to test and integrate the changes on my end

KaffeinatedKat commented 8 months ago

It calculates Sin and Cos (these are notoriously slow)

It does not appear to be a rendering time issue, added a simple lookup table for Sin and Cos for the second hand, which eliminates the slowness of these functions, and the screen still blanks out randomly. Gunna have to take a look at how this is rendered. Perhaps it's trying to render something out of bounds, or something of the likes

FintasticMan commented 8 months ago

Yeah, sin and cos are already implemented as a lookup table in LVGL.

mark9064 commented 8 months ago

RE: https://github.com/InfiniTimeOrg/InfiniTime/pull/1869#issuecomment-1776021839 The constant frequency frames work exactly as expected, I've verified this by keeping 60Hz while in always on mode as a test. I can't seem to find anything that displays frames stably, at least with the *8 divider. I guess I'll try some other dividers.

As a sidenote, the frame rate control command (B3) doesn't seem to work how the datasheet says. It says if FRES is clear the idle frame rate is defined by the normal mode frame rate, which in testing isn't true. It also says that the frame rate applies only to idle mode, which also isn't true. So I honestly have no clue what B3 does compared to C6? Did you figure anything out here

Edit: turns out the display does not refresh at the refresh rates given by the table, instead there's also a formula that gives the exact refresh rate. This discrepancy might be the issue, so I'll try with that in mind

mark9064 commented 8 months ago

I've configured it to refresh at exactly 32Hz/4Hz(div 8) (using the formulas), but instead it's refreshing at 2Hz. No clue why, I'm certain I've got my calculations using the refresh rate formula correct. Anyway it works great and battery life seems better, frame pacing is solid. Just need to figure out what's actually happening refresh wise now

KaffeinatedKat commented 8 months ago

Great to hear that, can't wait to merge those changes, good work so far. Interested to see if that happened to fix the analog watch face issue, as that could potentially have something to do with the inconsistent pacing

mark9064 commented 7 months ago

Still working on this (been busy with work) Got a bit sidetracked with some changes to time handling, there's currently quite a bit of jitter which causes the 2Hz display to occasionally miss seconds. I want to take the time to refactor things properly, so I'm probably going to have to create a couple other PRs which this one will depend on (as I don't want to include too many unrelated changes). This branch can pull them early though, so it doesn't block dev here

Re the disappearing display issue on wake: I tried navigating to the quick settings menu and switching to sleep mode while the display was blanked (as sleep mode disables AOD). Took a few tries to hit it as I was blind, but switching off and back on again (just with power button) after that brought the display to life (though with inverted colours). Curious to hear if this hack works for you

I've also tightened down all of the AOD display command timings with no ill effects, so the watch is much more responsive now when it comes to leaving/entering AOD. Will try get patches up soon but my branch is a bit of a mess lol