Please add check() and possibly gc() to SPIFFS/filesystems

[joysfera]

I'm experiencing SPIFFS issues (like fopen(name, "w") failing) and also SPIFFS eventual corruptions that could hopefully be fixed by calling SPIFFS.check() or possibly SPIFFS.gc()

These two functions exist on Arduino core for ESP8266. Please add them to this Arduino core for ESP32* as well.

FYI, the SPIFFS.check() seems to be implemented internally as follows: SPIFFS_check(&_fs);

The SPIFFS.gc() then simply calls SPIFFS_gc_quick(&_fs, 0);

Thank you!

[atanisoft]

@joysfera Please submit a PR that adds these methods so it can be reviewed.

[joysfera]

The SPIFFS is included just as a compiled library, I couldn't find its source code, so I don't know how to exactly implement that. Maybe the ESP IDF needs to export some functions first? Dunno.

[atanisoft]

The underlying SPIFFS code can be found here. It does appear that ESP-IDF would need to expose some functions that internally call check/gc/etc.

I'd also suggest moving over to LittleFS as SPIFFS is deprecated on most platforms (including ESP8266 from what I understand) for various performance/longevity issues.

[joysfera]

I'd like to move over to LittleFS if only I knew how to do it safely with devices deployed in the field having many valuable data files on the SPIFFS. Seems like the only safe way would be to repartition it to use OTA partition as a temporary data partition, format that as LittleFS, copy over all files, format the original data partition as LittleFS, copy all files from temporary data partition to the original one and repartition back that temporary data partition as OTA one. Or is there a better way?

[atanisoft]

I'm not sure there is a better way to migrate files from SPIFFS -> LittleFS other than what you describe, I'm also not 100% certain that it would work but it is worth experimenting with.

[VojtechBartoska]

Hello @joysfera, any news on this from your side? :)

[joysfera]

Sorry, I don't have a PR ready (and won't have in any near term).

[igrr]

Since SPIFFS library is a wrapper over esp-idf spiffs component, we would have to expose GC and check functions there. We have merge requests open for both, GC is ready to be merged, check needs a bit more work (internal: !15565, !17638). So I suggest we keep this open for now, it can be implemented pretty easily once IDF MRs are merged.

[VojtechBartoska]

@lucasssvaz Can you please check current status of this?

Take a look on ESP-IDF updates regarding this topic and check if there are any updates and if this is now doable, thanks.

It is lower priority, found this ticket during backlog clean-up. Thanks

[lucasssvaz]

@VojtechBartoska Both are now doable but the garbage collection function will require to configure CONFIG_SPIFFS_GC_MAX_RUNS based on the use case:

/**
 * @brief Perform garbage collection in SPIFFS partition
 *
 * Call this function to run GC and ensure that at least the given amount of
 * space is available in the partition. This function will fail with ESP_ERR_NOT_FINISHED
 * if it is not possible to reclaim the requested space (that is, not enough free
 * or deleted pages in the filesystem). This function will also fail if it fails to
 * reclaim the requested space after CONFIG_SPIFFS_GC_MAX_RUNS number of GC iterations.
 * On one GC iteration, SPIFFS will erase one logical block (4kB). Therefore the value
 * of CONFIG_SPIFFS_GC_MAX_RUNS should be set at least to the maximum expected size_to_gc,
 * divided by 4096. For example, if the application expects to make room for a 1MB file and
 * calls esp_spiffs_gc(label, 1024 * 1024), CONFIG_SPIFFS_GC_MAX_RUNS should be set to
 * at least 256.
 * On the other hand, increasing CONFIG_SPIFFS_GC_MAX_RUNS value increases the maximum
 * amount of time for which any SPIFFS GC or write operation may potentially block.
 *
 * @param partition_label  Label of the partition to be garbage-collected.
 *                         The partition must be already mounted.
 * @param size_to_gc       The number of bytes that the GC process should attempt
 *                         to make available.
 * @return
 *          - ESP_OK on success
 *          - ESP_ERR_NOT_FINISHED if GC fails to reclaim the size given by size_to_gc
 *          - ESP_ERR_INVALID_STATE if the partition is not mounted
 *          - ESP_FAIL on all other errors
 */
esp_err_t esp_spiffs_gc(const char* partition_label, size_t size_to_gc);