Status of nvs on (external) spi-flash (IDFGH-13530)

[Teesmo]

Answers checklist.

• [X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• [X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• [X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

I am working on an application that would store time-series entries, that I would like to query (prompting from a web-based application) using a key, or index associated to the data I want to access. The data has to be persisted over a very long period (>10 years), and new entries would be generated every 30 minutes. My initial thoughts are, that using nvs to store this data would be ideal, but nvs is currently confined to internal flash. The latest documentation suggests that a future implementation of nvs may support writing to (external) spi-flash. What is the current status of this development?

[KonstantinKondrashov]

Hi @Teesmo!

I am not an expert in the nvs component but I think the current version (at least the master) already allows it. You need to connect an external flash (spi_external_flash), register there an NVS2 partition, then initialize the nvs with this partition (nvs_flash_init_partition("nvs2")). Seems to me it should work, please test it first.

     const esp_partition_t *nvs2_part;
    esp_partition_register_external(spi_external_flash, nvs2_offset, nvs_part->size, "nvs2", ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_NVS, 0, &nvs2_part));
    nvs_flash_init_partition("nvs2");
    nvs_handle_t my_handle2;
    nvs_open_from_partition("nvs2", "storage", NVS_READWRITE, &my_handle2);
    uint32_t magic_value = 0x0729FEED;
    TEST_ESP_OK(nvs_set_u32(my_handle2, "magic_value", magic_value));
    TEST_ESP_OK(nvs_commit(my_handle2));
    // ....
    nvs_close(my_handle2);
    nvs_flash_deinit_partition("nvs2");

    // deregister "nvs2"
    esp_partition_deregister(nvs2_part);

[igrr]

Addition to @KonstantinKondrashov's comment:

You can follow the approach of ext_flash_fatfs example to connect the external flash and define a partition there:

https://github.com/espressif/esp-idf/blob/dbce23f8a449eb436b0b574726fe6ce9a6df67cc/examples/storage/ext_flash_fatfs/main/ext_flash_fatfs_example_main.c#L65-L73

Then, instead of mounting FATFS, you can initialize NVS in the partition you have defined, using nvs_flash_init_partition and nvs_open_from_partition function, similar to Konstantin's example above.

---

That being said, I think NVS is not going to work for this use case. If you store one key-value pair every 30 minutes, over 10 years you will generate around 175200 key-value pairs, or 1390 flash sectors, or around 5561 kB. Such a large NVS partition will require a lot of time to initialize. Additionally, according to this RAM usage estimate from the docs (Large Amount of Data in NVS,

For RAM usage estimation, please use the following approximate figures: each 1 MB of NVS flash partition consumes 22 KB of RAM and each 1000 keys consumes 5.5 KB of RAM.

Near the end of the 10 year period, the RAM usage of NVS may reach >100kB. Newer IDF versions have an option to allocate NVS cache in PSRAM, so this extra RAM usage may or may not be an issue, depending on whether you are using PSRAM or not. However, the initialization time of NVS for such a partition will still be quite long:

Initialization of NVS requires approximately 0.5 seconds per 1000 keys.

175200 * 0.5 / 1000 = 87.6 seconds.

Therefor I think for your use case either a conventional filesystem or a custom implementation based directly on esp_partition API might be more suitable.

[Teesmo]

Thank you for the insights. Are you suggesting that I should rather use FATFS/LittleFS as it would work faster? I should be honest, I don't how file search works in file systems (I know very little about file systems, in fact).