TSDB without rollover corrupts database if all sectors are inuse

[Andste82]

Hi, i have a TSDB running on an ESP32 and i have disabled the rollover mechanism to keep my old data instead of overwriting it with new entries. When the flash partition is full, fdb_tsl_append returns with FDB_SAVED_FULL which is OK. But if i reboot my device, the database seems to be broken:

TSDB (log) oldest sectors is 0x00000FFF, current using sector is 0xFFFFFFFF.

And when i then try to read data, i receive the following error message:

[E/FAL] (fal_partition_read:415) Partition read error! Partition address out of bound.

My expected behaviour would be that the database will not accept new data until enough old data has been marked as deleted, so when new data is added, the freed sector can be deleted and reused. But i can't read the old data so cannot mark them as deleted.

[armink]

What version are you using

[Andste82]

You're latest release: FlashDB V1.1.2 is initialize success.

[armink]

Can you confirm that the code on line 784 is executed?

https://github.com/armink/FlashDB/blob/91072137d7e9cfe519053c481a266f913d085eeb/src/fdb_tsdb.c#L777-L786

[Andste82]

I actually cannot execute the programm, but i'm very sure that this line is not executed!

The reason is the following code, which i find very strange: Function fdb_tsdb_control @ line 697

    case FDB_TSDB_CTRL_SET_ROLLOVER:
        /* this change MUST after database initialized */
        FDB_ASSERT(db->parent.init_ok == true);
        db->rollover = *(bool *)arg;
        break;

because of this assertion i set the rollover to false after fdb_tsdb_init, so the code you mentioned will surely not be executed

[armink]

OK, The FDB_TSDB_CTRL_SET_ROLLOVER shoud before initialization. Can you change it and try again?

    case FDB_TSDB_CTRL_SET_ROLLOVER:
        /* this change MUST before database initialization */
        FDB_ASSERT(db->parent.init_ok == false);
        db->rollover = *(bool *)arg;
        break;

[Andste82]

Now i was able to test it again. Your changes only half work, because fdb_tsdb_init sets the rollover back to the default value true.

I also removed line 758, and than it seems to work:

    db->get_time = get_time;
    db->max_len = max_len;
    /* default rollover flag is true */
    // db->rollover = true;
    db->oldest_addr = FDB_DATA_UNUSED;
    db->cur_sec.addr = FDB_DATA_UNUSED;
    /* must less than sector size */
    FDB_ASSERT(max_len < db_sec_size(db));

If the partition now gets full and i reboot the system, the database is still valid and all data can be read ...

But there is now another problem:

In case the database gets full (so fdb_tsl_append returns FDB_SAVED_FULL) i walk over every database entry and tag each entry with the flag FDB_TSL_DELETED to tell the TSDB that every sector can now be deleted and reused. But the Database won't delete anything, every try to append new data will still abort with FDB_SAVED_FULL, which means that the database cannot store data anymore!

[Iris74123]

Please tell me how the follow-up was solved, I also encountered the same problem.

[Andste82]

@Iris74123 There is currently no solution for this problem. I simply implemented a limit in the application that ensures that the flash never gets full - not perfect, but it works.

[Iris74123]

@Andste82 Okay, thank you very much!