espressif / esp-idf

Espressif IoT Development Framework. Official development framework for Espressif SoCs.
Apache License 2.0
13.31k stars 7.2k forks source link

NVS manual garbage collection (IDFGH-13397) #14305

Open bryghtlabs-richard opened 1 month ago

bryghtlabs-richard commented 1 month ago

Is your feature request related to a problem?

When NVS has plenty of erased space, writes are fast. But after NVS has been fully written writes require erases of previously written data, which make my UI stutter.

Describe the solution you'd like.

I would like the ability to request garbage collection and erasure of free space at a time when it makes sense for my application, similar to TRIM command for managed Flash SSDs. Once complete, most unused space in NVS would be erased.

Describe alternatives you've considered.

I've considered deferring writes or commits to NVS, but this is not a great workaround, as it increases the time where data may be lost before NVS writes to Flash.

Additional context.

It might be nice to be able to specify a limit in terms of time.

rrtandler commented 1 month ago

Hi @bryghtlabs-richard,

The different response of NVS api in the "empty" vs "full" scenarios are result of different way, the free space for the newly written data is discovered. If the NVS has already written into all the 4k pages reserved to it, subsequent attempts to write the data can either fit into the remaining space in the active NVS page (then it is fast) or needs to reclaim some space from the NVS pages marked as full (with some entries in it marked as erased). Then the "local" defragmentation takes place as a part of the nvsset* operation. Which, because of its unpredictability, causes annoying delays.

If I understand your proposal, you would like to extend the API by function defragmenting the complete NVS space. In the context of this proposal, what exactly is meant by the note: "It might be nice to be able to specify a limit in terms of time." ?

Please share also more about the granularity of data storage (whether you use blob or store individual items using single entry operations) and frequency of updates.

bryghtlabs-richard commented 1 month ago

Hi @rrtandler, you understand my problem well.

In the context of this proposal, what exactly is meant by the note: "It might be nice to be able to specify a limit in terms of time." ?

Optional partial defragmentation. On startup, I'd like to defrag the whole partition. But once our UI is running, sometimes there are better times to defrag that are not be long enough to defrag the entire partition, then I might like to request defragmentation, but once defragmentation exceeds the specified time, please return soon.

Please share also more about the granularity of data storage (whether you use blob or store individual items using single entry operations) and frequency of updates.

We have a 64KB NVS, about 25% utilization on most devices, mix of items: