search

armink / FlashDB

An ultra-lightweight database that supports key-value and time series data | 一款支持 KV 数据和时序数据的超轻量级数据库
Apache License 2.0
1.7k stars 400 forks source link

V2.1.0使用tsdb遇到一个问题，假如tsdb刚好全部写满的时候重启了，再次开机 初始化tsdb的时候，会出现oldest sectors is 0x00000FFF, current using sector is 0xFFFFFFFF， 导致后续无法写入 #291

Open Curdle opened 2 months ago

Curdle commented 2 months ago

写个小demo验证此问题：

1、先将fdb_tsdb_init增加rollover传参，方便做写满测试 fdb_err_t fdb_tsdb_init(fdb_tsdb_t db, const char *name, const char *path, fdb_get_time get_time, size_t max_len, void *user_data, bool rollover) { ... // db->rollover = true; db->rollover = rollover; ... }

2、使用如下代码测试 ，先将tsdb写满，然后重新初始化，就会出现oldest sectors is 0x00000FFF, current using sector is 0xFFFFFFFF， 导致后续无法写入

int tsdb_test(void) { // 第一次初始化，将rollover置false，为的是将tsdb写满，模拟问题场景 flashdb_tsdb_init(false); fdb_tsl_clean(&tsdb1); char data[128] = {0}; memset(data, 0x11, 128);

// 开始写入，将tsdb写满
do {
    fdb_err_t err = flashdb_tsdb_write(data, 127);
    if (err != FDB_NO_ERR) {
        printf("%s write err = %d\n", __func__, err);
        break;
    }
    vTaskDelay(10/portTICK_PERIOD_MS);
} while (1);

// 写满后重新初始化，将rollover置true，此时为正常使用场景
fdb_tsdb_deinit(&tsdb1);
printf("%s reinit\n", __func__);
flashdb_tsdb_init(true);

// 再次写入测试
do {
    fdb_err_t err = flashdb_tsdb_write(data, 127);
    printf("write err = %d\n", err);
    vTaskDelay(2000/portTICK_PERIOD_MS);
} while (1);
return 0;

}

打印日志如下：

[2024/4/17 11:11:07-248] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:07-249] check_sec_hdr_cb sector->check_ok = 1, sector->status = 2, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:07-249] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-249] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-249] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-249] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-249] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-249] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-249] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-249] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-250] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-250] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-250] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-250] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-250] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-250] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-250] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0x1000 [2024/4/17 11:11:07-250] [FlashDB][tsl][log][fdb_tsdb1] (../components/FlashDB/src/fdb_tsdb.c:1014) TSDB (log) oldest sectors is 0x00000000, current using sector is 0x00010000. [2024/4/17 11:11:07-250] [FlashDB] FlashDB V2.1.0 is initialize success. [2024/4/17 11:11:07-250] [FlashDB] You can get the latest version on https://github.com/armink/FlashDB . [2024/4/17 11:11:07-758] [FlashDB][tsl][log][fdb_tsdb1] All sector format finished. [2024/4/17 11:11:15-071] [FlashDB][tsl][log][fdb_tsdb1] Error: update the sector status failed (7) [2024/4/17 11:11:15-071] flasbdb_test write err = 7 [2024/4/17 11:11:15-071] flasbdb_test reinit [2024/4/17 11:11:15-071] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-071] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-072] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-072] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-072] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-072] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-072] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-072] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-072] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-072] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-072] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-073] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-073] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-073] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-073] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-073] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-073] check_sec_hdr_cb sector->check_ok = 1, sector->status = 3, db->cur_sec.addr = 0xffffffff [2024/4/17 11:11:15-073] fdb_tsdb_init 111 [2024/4/17 11:11:15-073] fdb_tsdb_init 222 [2024/4/17 11:11:15-073] [FlashDB][tsl][log][fdb_tsdb1] (../components/FlashDB/src/fdb_tsdb.c:1014) TSDB (log) oldest sectors is 0x00000FFF, current using sector is 0xFFFFFFFF. [2024/4/17 11:11:15-073] sector->magic: 0x4c5354ff, check fail, addr: 0xffffffff [2024/4/17 11:11:15-073] [FlashDB][tsl][log][fdb_tsdb1] Error: update the sector status failed (7) [2024/4/17 11:11:15-073] write err = 7 [2024/4/17 11:11:17-063] [FlashDB][tsl][log][fdb_tsdb1] Error: update the sector status failed (7) [2024/4/17 11:11:17-063] write err = 7 [2024/4/17 11:11:19-064] [FlashDB][tsl][log][fdb_tsdb1] Error: update the sector status failed (7) [2024/4/17 11:11:19-064] write err = 7 [2024/4/17 11:11:21-066] [FlashDB][tsl][log][fdb_tsdb1] Error: update the sector status failed (7) [2024/4/17 11:11:21-066] write err = 7 [2024/4/17 11:11:23-062] [FlashDB][tsl][log][fdb_tsdb1] Error: update the sector status failed (7) [2024/4/17 11:11:23-062] write err = 7 [2024/4/17 11:11:25-064] [FlashDB][tsl][log][fdb_tsdb1] Error: update the sector status failed (7) [2024/4/17 11:11:25-064] write err = 7 [2024/4/17 11:11:27-064] [FlashDB][tsl][log][fdb_tsdb1] Error: update the sector status failed (7) [2024/4/17 11:11:27-064] write err = 7 [2024/4/17 11:11:29-066] [FlashDB][tsl][log][fdb_tsdb1] Error: update the sector status failed (7) [2024/4/17 11:11:29-066] write err = 7

Curdle commented 2 months ago

暂时我的处理是在sector_iterator()增加一道检查，扫完全部sector依旧是FDB_SECTOR_STORE_FULL的情况下，就将0地址的sector给erase，但会导致tsl_format_all()的时候，对0地址多erase了一次

static void sector_iterator(fdb_tsdb_t db, tsdb_sec_info_t sector, fdb_sector_store_status_t status, void arg1, void arg2, bool (callback)(tsdb_sec_info_t sector, void arg1, void *arg2), bool traversal) { uint32_t sec_addr = sector->addr, traversed_len = 0;

/* search all sectors */
do {
    read_sector_info(db, sec_addr, sector, false);
    if (status == FDB_SECTOR_STORE_UNUSED || status == sector->status) {
        if (traversal) {
            read_sector_info(db, sec_addr, sector, true);
        }
        /* iterator is interrupted when callback return true */
        if (callback && callback(sector, arg1, arg2)) {
            return;
        }
    }
    traversed_len += db_sec_size(db);
} while ((sec_addr = get_next_sector_addr(db, sector, traversed_len)) != FAILED_ADDR);
// FDB_DEBUG("TSDB (%s) sec_addr is 0x%08" PRIX32 ", traversed_len is 0x%08" PRIX32 ".\n", db_name(db), sec_addr, traversed_len);  
if (sec_addr == FAILED_ADDR && traversed_len >= db_max_size(db) && sector->status == FDB_SECTOR_STORE_FULL) {

    FDB_DEBUG("TSDB (%s) all sectors are full and need to be reset \n\n", db_name(db));  
    db->cur_sec.addr = 0;
    format_sector(db, db->cur_sec.addr);
    read_sector_info(db, db->cur_sec.addr, &db->cur_sec, true);
    return;
}

}