search

InfiniTimeOrg / InfiniTime

Firmware for Pinetime smartwatch written in C++ and based on FreeRTOS
GNU General Public License v3.0
2.75k stars 940 forks source link

External flash usage #321

Closed Avamander closed 2 years ago

Avamander commented 3 years ago

It would be good to map out where LittleFS would potentially be used in InfiniTime, right now and in the future. What would have to be migrated and what can't be migrated.

Current usages of flash contain:

Future usages might contain:

joaquimorg commented 3 years ago

I managed to do a first performance test between RAWFS versus LITTLEFS. The use of LITTLEFS adds another 20Kb in the firmware. As we can see in the video the performance between the two is evident.

Pinetime Lite - RAWFS vs LITTLEFS

The LITTLEFS introduces some advantages in the management of the files, but I think that the loss of performance and increase of the firmware does not justify the gain of functionalities.

In my case, I will not invest more time in the littlefs solution, I will try to improve RAWFS and add some features to allow partial updates or subdivide between resources and watch faces, so that I can have customizable watch faces without having to always send everything.

Avamander commented 3 years ago

What's the LittleFS configuration you used?

joaquimorg commented 3 years ago

What's the LittleFS configuration you used?

https://github.com/joaquimorg/PinetimeLite/blob/1db8712ecd49ecb2ee37147eebe2e5b6dac521ac/src/components/fs/FS.cpp#L81

Avamander commented 3 years ago

I will try to optimize the settings, because I think better read performance is doable and worth the investment.

The advantages we'd get from LittleFS, such as reduced maintenance and development burden, full-flash wear leveling, better failure tolerance (CoW), thought-out and documented design, higher interoperability and alternative implementations are more than some.

joaquimorg commented 3 years ago

In my tests I had already identified that LVGL is always opening files to read, and did not keep the files open while refreshing the screen. After all, it was my mistake, the code allow that there could be several files open at the same time, but LVGL does not take advantage of this if it is not indicated in LV_IMG_CACHE_DEF_SIZE how many images we want to keep open. I will make this change and carry out further tests.

I would also prefer to use littlefs, but I don't want its use to translate into having something that makes a point of losing the little performance we have.

When I indicated that I would not invest more time in littlefs it was in the continuation of developing the mechanisms so that we can have a management of the files, delete files, create folders etc...

So whenever I discover something that can improve the performance I will test it.

JF002 commented 3 years ago

Thanks @joaquimorg and @Avamander , great work!

From your video, the performance penality looks... surprisingly huge! Maybe @Avamander will be able to fine-tune/optimise the integration of littlefs with lvgl?

Now... Maybe we are trying to solve many problems with a single solution? The use-case tested by @joaquimorg on his video is a "read-only" use case : we store files in the external memory once and only read them from the firmware. Those data are (over)written only when upgrading the graphical assets. The use-case where a more advanced file system like littlefs would bring a lot of advantages are "read/write" use-cases like storing 24h or heart rate data, user settings, save the master piece the user's just drawn using InfiniPaint,...

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

What's your opinion?

Avamander commented 3 years ago

Thanks joaquimorg and Avamander , great work! Maybe Avamander will be able to fine-tune/optimise the integration of littlefs with lvgl?

joaquimorg deserves more credit here and he also said that LV_IMG_CACHE_DEF_SIZE might provide a quick and easy way of regaining the lost performance, if that doesn't work out I'll investigate as well.

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

Yes, that's a theoretical possibility. But the returns are somewhat smaller. If it still turns out that LFS is not worth it, we just have to go with an alternative solution.

joaquimorg commented 3 years ago

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

I think it may be a viable solution, a few days ago we were discussing this on the discord.

Really the part of the resources for the firmware can be read-only and we don't have the need to have littlefs for that, and we can have an r/w zone with littlefs, in total we have about 3.4 Mb still free if we share between the two we have space for lots of applications to save data.

In the end we have to remember that we are talking about a device that has its limitations and we do not want to have a very complex system, my goal is always to look for a way to try to do things to get the maximum performance from the smartwatch which it often leads to doing things in a less generic way, more oriented towards the optimization of the few resources we have.

This weekend I will try to do some more tests to try to see if the changes have influence.

ObiKeahloa commented 3 years ago

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

We can also make a proper filesystem with folders (Directory's) , Example: A fonts folder for the font files , Assets that have all assets for the firmware (Resources) , AppData (Contains appdata that is stored to the spi flash) , Userdata (User Settings , Time Settings etc,)

If possible we could also try to make the Assets r/w , so that the user can add or modify the assets without having to make an entirely new firmware....

This could be a vulnerability but a hierarchy system can be put in place: System > User > Apps. This might need a lot of code so might be out of reach.

joaquimorg commented 3 years ago

I did some more tests changing the LVGL configuration (LV_IMG_CACHE_DEF_SIZE) and it really made all the difference, of course there is some loss of performance, but I think it is possible to live with it. Now the hardest work begins, turning everything into a PR ...

Pinetime Lite - LITTLEFS

Avamander commented 3 years ago

I'm amazed, truly incredibly cool. Thanks @joaquimorg

JF002 commented 3 years ago

This is really impressive! Thanks for your experimentation!

So, LVGL could be the unique answer for all our "external" storage use-cases?

For the PR, what would be the easiest way for you? A single PR of split the changes accross multiple PR to ease the review process? Do you have a branch on github so we can have a look at how it works?

joaquimorg commented 3 years ago

Do you have a branch on github so we can have a look at how it works?

The branch with the test code is here https://github.com/joaquimorg/PinetimeLite/tree/UsingLITTLEFS

joaquimorg commented 3 years ago

For the PR, what would be the easiest way for you? A single PR of split the changes accross multiple PR to ease the review process?

I will try to split as much as possible to be easier to include without causing breaks in what already exists, and thus also be easier to review, however it will only work in full when everything is together.

JF002 commented 3 years ago

I will try to split as much as possible to be easier to include without causing breaks in what already exists, and thus also be easier to review, however it will only work in full when everything is together.

Awesome! Feel free to comment here or to ping us in the chat if you need help!

ObiKeahloa commented 3 years ago

I did some more tests changing the LVGL configuration (LV_IMG_CACHE_DEF_SIZE) and it really made all the difference, of course there is some loss of performance, but I think it is possible to live with it. Now the hardest work begins, turning everything into a PR ...

Pinetime Lite - LITTLEFS

Awesome! Really nice that it has possibilities to be speedy.

JF002 commented 3 years ago

@Avamander provided this link in the chat room, it could be interesting to have a closer look at this lib/protocol : https://github.com/adafruit/adafruit_circuitpython_ble_file_transfer

JF002 commented 3 years ago

@joaquimorg Any news on this topic?

I was thinking : how will we use this filesystem? I mean, we have multiple types of data :

I have many questions about this, and I think you've already solved most of them in PineTimeLite :)

User settings and runtime data will be created and read/written by the firmware.

I'm more concerned about the resources : The user will need to manually flash (OTA) these data for the FW to work correctly. What happens if the user does not send the file? Is there an error? Or a message asking the user to flash the data? Or the FW works in "degraded" mode without the nice pictures?

Also, what protocol do we use to send the resources? DFU? A custom one? Another one (like the one from adafruit mentioned by Avamander) ?

Next : how do we generate these data? I guess you have a script that converts pictures into that binary file? Do you also handle versioning so that the firmware can check if the correct resources file is installed ?

These questions are mostly implementations details, but I think there are many many ways to do all of this, and I would like to have your opinions about this :)

ObiKeahloa commented 3 years ago

@joaquimorg Any news on this topic?

I was thinking : how will we use this filesystem? I mean, we have multiple types of data :

  • read only data for the firmware (I think you call them resources in your FW): logo, icons, pictures,...
  • User settings
  • "run time data" like HR and steps values

I have many questions about this, and I think you've already solved most of them in PineTimeLite :)

User settings and runtime data will be created and read/written by the firmware.

I'm more concerned about the resources : The user will need to manually flash (OTA) these data for the FW to work correctly. What happens if the user does not send the file? Is there an error? Or a message asking the user to flash the data? Or the FW works in "degraded" mode without the nice pictures?

Also, what protocol do we use to send the resources? DFU? A custom one? Another one (like the one from adafruit mentioned by Avamander) ?

Next : how do we generate these data? I guess you have a script that converts pictures into that binary file? Do you also handle versioning so that the firmware can check if the correct resources file is installed ?

These questions are mostly implementations details, but I think there are many many ways to do all of this, and I would like to have your opinions about this :)

It could show an error saying "Resources not found" similar to the mi band that says "Please connect to mi fit"

We could use the same method we are using now for this , the resources can be a compressed file that is uncompressed once sent to the watch.

joaquimorg commented 3 years ago

@joaquimorg Any news on this topic?

Hi, unfortunately I haven't been able to dedicate much time to the project, my work has taken up a lot of time.

I was thinking : how will we use this filesystem? I mean, we have multiple types of data :

  • read only data for the firmware (I think you call them resources in your FW): logo, icons, pictures,...
  • User settings
  • "run time data" like HR and steps values

I have many questions about this, and I think you've already solved most of them in PineTimeLite :)

User settings and runtime data will be created and read/written by the firmware.

I'm more concerned about the resources : The user will need to manually flash (OTA) these data for the FW to work correctly. What happens if the user does not send the file? Is there an error? Or a message asking the user to flash the data? Or the FW works in "degraded" mode without the nice pictures?

In my fork I do not present any information, what happens is that the images do not appear, however it would be good to present some information about the lack of updating the resources, to avoid that the user does not think that the update was not done correctly.

Also, what protocol do we use to send the resources? DFU? A custom one? Another one (like the one from adafruit mentioned by Avamander) ?

I do not think that DFU can be used, since it only allows to have the Bootloader, Softdevice and Application in the payload, so I was able to investigate. So we would have to have something to be able to upload the resources. It would be useful to be able to upload together or separately.

The adafruit example is very similar to the one I have, mine is a little simpler but it works. If we want to have a way to be able to control the files that are in the flash we will have to have an additional service to the DFU, in order to be able to list, create and delete files.

Next : how do we generate these data? I guess you have a script that converts pictures into that binary file? Do you also handle versioning so that the firmware can check if the correct resources file is installed ?

Yes I have a script that converts PNG to LVGL images, then another script creates the RAWFS file, it is this file that will be sent to the watch. And yes the RAWFS has the version in the header, to be able to validate that it is the right one when the watch starts.

However with littlefs I think it cannot be done the way I have it, in the tests I did the files were all separate, and this is the way that makes the most sense to me, for example having a "resources" folder and inside the necessary images, there will also be a metadata file in that folder with the version information and other information that is necessary to identify that the resources are valid for the version that the watch is running.

The generation of resources can be done by generating a zip with all the necessary files, and it is on the side of the companion application to open that zip and send each file to the watch inside the "resources" folder.

ObiKeahloa commented 3 years ago

The generation of resources can be done by generating a zip with all the necessary files, and it is on the side of the companion application to open that zip and send each file to the watch inside the "resources" folder.

Much better than my idea of senting a compressed file , will be less taxing on us and the watch.

JF002 commented 3 years ago

@joaquimorg Thanks for all these informations, and I fully you prioritize your job on this project :)

I don't think we have a use-case for a "file explorer" API right now but we might need a way to remove older resources when we upgrade to a new version to avoid filling the memory with data that are not needed anymore.

I do not think that DFU can be used, since it only allows to have the Bootloader, Softdevice and Application in the payload

Technically, we might be able to use the DFU protocol, as the field for the data type is 1 byte long and only 3 values are defined by NRF (other values are "reserved"), so we could use those reserved values to add the resource type, but that wouldn't be "nrf" compliant. But, as you said, dfu does not provide any way to control the files (to delete older/unused files, for example).

So, if I understand correctly, we have to implement

joaquimorg commented 3 years ago

Base support for littlefs added in https://github.com/JF002/InfiniTime/pull/438

doniks commented 2 years ago

It would be good to map out where LittleFS would potentially be used in InfiniTime [...]

  • [ ] ??? Please leave your comment below

Replace the manual FW validation with an automatic one:

So, if the FW update is not good, it would try to start, fail three times and then roll back.

No need for the user to know about and understand the manual validation and dig around in the settings to find the place.

(*) Don't rollback after only 1 failed boot. Someone, sometime will turn it off right after turning on, or it will fail due to low battery. That doesn't mean the FW should be rolled back

Avamander commented 2 years ago

@doniks actually there is a need, it's verifying the hardware works. That can't be done software-side. There's also no need to dig if the quick start guide has been read.

doniks commented 2 years ago

@doniks actually there is a need, it's verifying the hardware works. That can't be done software-side. There's also no need to dig if the quick start guide has been read.

When you say whether "the hardware works" do you actually mean whether the hardware itself broke for some reason? I guess not, because actually if the hardware broke, then rolling back the FW won't help either. So, I assume we are talking about "hardware support", ie, some firmware issue.

I maintain that it's a better user experience to only roll back if the FW can't run at all, ie the device is a brick unless we roll back. As long as the FW does start (and the user is able to perform another update/downgrade to some other firmware version) then I think it's a better user experience to keep the new version by default, rather than to roll back by default. Even if the new FW has some serious issue like it can't read the accelerometer anymore. You wouldn't assume that every update is broken unless the user explicitly confirms that this time around it's ok. What must be prevented is that some bad update leads to endless boot-crash cycles and 'normal' users are unable to fix it. But once you have avoided that, you wouldn't want to push too much 'maintenance' hassle onto the user

ialokim commented 2 years ago

Base support for littlefs added in #438

  • [x] the filesystem
  • [ ] the protocol to send data to this filesystem
  • [ ] the file format of the resources + versioning
  • [ ] using those resources in the firmware and handling the case when the resources are not available.

If I investigated correctly, #756 should be the second bullet point, right? I would like to help working on the next step to get the external memory usable for resources, but I just wanted to ensure I didn't overlook some work that has already been put into specifying the file format / directory structure for the resources. If no one objects, I would open a new issue with some proposals.

Just as a recap, what kind of resources do you want to support? I can think of:

JF002 commented 2 years ago

You are right, #756 is the 2nd bullet! So now, we have the filesystem and an API to send data to this file system from a companion app. We currently haven't specified any file format or directory structure so your analysis and proposals are more than welcome!

The list of resources you mentioned looks quite right. Note that we don't need to support all of them in one shot. We can start with only one type of data, the one we'll find the easiest to implement, and add the other ones later one.

For example, for the December Pine64 community update, I (with the help of @geekbozu) wrote a very simple change in the Digital watch face to display a background image if it's available on the file system. I think that would be a nice first step to implement.

Note that we'll probably want to add a bullet point : "improve the performance of the SpiNorFlash driver and/or fileesystem integration", as reading a full picture from the filesystem slows down the whole UI.

geekbozu commented 2 years ago

So something to note, A lot of the slowdown has to do with how often LVGL opens the image and how much of it it can cache. Since LVGL is "blitting" the image it has to read the image and seek in it every line...

A large speedup will be using properly "compressed" (Smaller Bit per pixel with pallet data) images on the SPI flash instead of the demo we put together which loaded uncompressed images.

That needs a LVGL Image Decoder To handle this "compressed" format.

JF002 commented 2 years ago

I've finally decided to focus on this feature. I've created a new project dedicated to this topic.

First, a did a few benchmark of a first use-case : read a picture and use it as a full screen background for the digital watchface.

First, using InfiniTime without any change to the FS layer. Vertical scroll looks good, and similar to @joaqimorg results on this video. However, left/right animation are veeeery slow.

https://user-images.githubusercontent.com/2261652/169712859-54941329-9395-4680-804c-2a22f4f2a6f4.mp4

I compared my results with PineTimeLite, @joaquimorg 's fork. Both animations are very fast. This implementation uses a "raw" FS layer instead of LittleFS.

https://user-images.githubusercontent.com/2261652/169712700-5cca2ccb-0148-4453-8d92-4e31f2353199.mp4

I integrated this raw fs layer in InfiniTime and observed similar results:

https://user-images.githubusercontent.com/2261652/169712687-a18e54c6-4107-48e9-aafb-5fdd571145d7.mp4

So, my results are consistent with the ones from @joaquimorg, great!

The current implementation with LittleFS (which is still our preferred one) is quite efficient for linear readings, but way too slow for random read accesses needed by the left/right animation. During the left/right animation, LFS seeks in the file in the correct positions and then reads 8 bytes. This translates to 2 to 8 read accesses on the SPI bus. I think this is LittleFS that is looking for the block that contains the data and this is probably what's causing this huge slowdown...

Next, I'll try other use-cases : read smaller pictures and icons, and also fonts. The goal is to check which use-cases could already be implemented and which ones need more work!

JF002 commented 2 years ago

Here's an update on the font test : loading font from the external flash memory works out of the box. The code is pretty simple : 

lv_font_t* font = lv_font_load("F:/font1.bin");
...
if(font != nullptr)
    lv_obj_set_style_local_text_font(label_time, LV_LABEL_PART_MAIN, LV_STATE_DEFAULT, font);
...
lv_font_free(font);

I grabbed 2 random fonts, converted them using lv_font_conv tool and sent them to my PineTime using ITD. Here's an example with a big font for the digital watchface: image

And a smaller font in the SystemInfo app: image

There's no visible impact on the performance of the display : everything is as fast as with the built-in fonts. Great! However, this is not so great regarding the RAM memory usage : when calling lv_font_load(), LVGL loads the whole font (header + font data) in memory. Which means that it allocates (lv_mem_alloc()) a few big buffers in memory.

Here are the memory usages of those fonts:

The memory currently available to LVGL is 14KB in total, so this leaves enough room to load 1 or 2 fonts at a time, depending on the current app running. However.. this is not always true due to memory fragmentation... If memory is fragmented and if it cannot find a memory block big enough to load the font, font loading will fail. I've also noticed that the digital watchface would crash because further allocations needed by the watchface would fail too...

The way LVGL loads the font in RAM is very good for performances, but it'll be a bit challenging in our memory constraint device :)

JF002 commented 2 years ago

New experiment : implement a new font engine that reads the data from the file in the SPI flash instead of reading the data (previously copied from flash to ram) from RAM memory.

This is a incomplete font engine. It does not dynamically read the font and glyph descriptions from the font data to read the glyph sat the correct position in the file. It just reads a fixed amount of bytes approximately at the position of the glyph. So the result is not as good looking as it should.

In this video, the big font for the time is read from the file:

https://user-images.githubusercontent.com/2261652/172709413-0a034fa6-8a22-4af6-b03a-4f6ddf050ca1.mp4

In this one, both the background and the font for the time are read from the file:

https://user-images.githubusercontent.com/2261652/172709454-a930f2f5-3443-446e-80f0-d1d78642ad08.mp4

As you can see, reading the font from the file slows down the scroll animation a little bit.

As I wrote above : this font engine is not complete and does not read all the data dynamically from the file. It also does not handle glyph buffers that are not aligned on 1 byte (if the buffer is not aligned on 1 bytes, LVGL reads data 1 byte at the time, see this). So these demo are probably a bit more optimistic than what we would get with a complete font engine...

nische commented 2 years ago

Hi,

I know that the ESP32 can load elf (binary-files) as libs from the SPI-Flash in RAM. Do you think that is possible too?

That could be a very nice way to Store Games and Features that the user dont need much.

BR, NiSche

JF002 commented 2 years ago

@nische Yes, it's definitely possible to do that with the PineTime too. However the ESP32 has 320KB or RAM and its SPI bus runs at 80Mhz, while the NRF52832 has only 64KB or RAM and a SPI bus at 8Mhz. Those limited resources makes the implementation of loadable apps a bit more challenging ;-)

nische commented 2 years ago

"a Bit" :😅

Maybe Like the Apps in Waspos. You get a List of all Apps and choose witch one will able to Run. If you uncheck a App the memory will released etc.

Maybe i find some Paper for the nrf5 Chips to get a easy entry in a soldution

JF002 commented 2 years ago

Waspos is written in Python. The interpreted language allows to easily load and unload modules. Such flexibility is more difficult to reach with compiled languages like C and C++.

JF002 commented 2 years ago

Here's a new "real life" example based on the G7710 watchface. In this video, the 3 fonts needed by the watchface are loaded in RAM at runtime when initializing the watchface.

https://user-images.githubusercontent.com/2261652/175815296-20ff3434-3e78-4e41-bf04-8cc2e7d1265e.mp4

As expected, the refresh of the display is quite fast as the fonts are effectively read from RAM. However, you can see that there's a bit of latency between the swipes and the beginning of the transition to the watchface. This is caused by the loading of the fonts from the external memory to the internal RAM.

Those fonts use ~8KB of RAM from the heap allocated to LVGL. Memory available in this heap goes from ~10-13KB to 2-4KB at run time.

JF002 commented 2 years ago

In the next video, I tried to workaround the very slow loading of watchface that use a full-screen bitmap as background (see here) : the loading of the background is deferred until the 1st frame, so that it's not done in the ctor() of the watchface.

https://user-images.githubusercontent.com/2261652/175816531-5d59c493-fc20-40cc-8491-661efb1a6e18.mp4

This ensures that the background will be read from the external memory in 1 chunk instead of several small chunks when it's read during the transition.

What do you think of the result?

Avamander commented 2 years ago

Looks great. I guess if the image is not rendered under the changing text, it wouldn't even be very noticeable.

JF002 commented 2 years ago

New test with Infineat watchface (logo and fonts loaded from external memory) :

https://user-images.githubusercontent.com/2261652/176535761-0499d1cc-b3d7-4187-8d53-070934df2e34.mp4

Fonts and logo from this comment, code based on the branch infineat-color by @dmlls.

JF002 commented 2 years ago

Let's put everything together : https://video.codingfield.com/w/swqopgt9p561ZBiTtxbTAf

I'm honestly quite impressed by the result!

Infineat by @dmlls G7710 by @ITCactus

JF002 commented 2 years ago

I pushed the code of the above demo in this PR. Feel free to test it and provide feedback (but read the warning first :)).

I tested it for a few days and enjoyed those new watchfaces ! The user experience seems quite good, and the loading of the big pictures could be improved by using more compressed (less colors) pictures.

At this point, I think that the performances are good enough to be used in InfiniTime. This new feature will allow to add a few more watchfaces, which will be more that appreciated by a lot of users (me included!).

But before rolling out this feature, we first need to figure out how to integrate it nicely in InfiniTime so that it's easy to use and error proof :

Zandengoff commented 2 years ago

How to upload it (we'll need the collaboration of companion app developers).

Do you know if anything has been added to the gadgetbridge issues list for this?

JF002 commented 2 years ago

@Zandengoff I've already contacted developers from Gadgetbridge and Amazfish, and they say that adding the feature should not be an issue. Now, I think it's up to us (InfiniTime developers) to describe the upload procedure with more details (how will the data be packaged, how to handle update of the resources, how to remove older versions,...) so companion app developers have all the necessary info to implement it.

Zandengoff commented 2 years ago

@JF002 Excellent, wanted to help with the request if needed, but seems you are already ahead of the curve.

JF002 commented 2 years ago

I'll close this issue/feature request as the external resource feature will be release in InfiniTime 1.11.Other topics (buffer for HR/Steps and other usage of the external flash memory) will be discussed in their dedicated issues.

Thanks everyone for your help on this huge topic!