Possible lost data and file corruption

[kmvanbrunt]

I am experiencing lost data and/or file corruption when using unqlite 1.1.8 on Windows. The following code reliably produces this behavior and has been tested on Windows 7 and 10.

The code inserts 200 records into a database and calls unqlite_commit() after each insertion because frequent commits seem to make the problem happen faster. After inserting the records, I count how many were inserted and the value is 200. I then close and reopen the database and recount the records. The new count is substantially lower (I've seen as low as 6).

Things I've noticed:

1. Removing the call to unqlite_commit() results in the correct amount of records being read back from the file.

2. Using a constant seed in srand() results in the same number of missing records each time. Therefore identical keys/records produces the same behavior every run. I have a commented seed for this purpose.

Am I doing anything wrong in this code?

#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <time.h>

#include "unqlite.h"

void randomBytes(uint8_t* bytes, size_t count)
{
    for (size_t i = 0; i < count; i++)
    {
        bytes[i] = (rand() % 256);
    }
}

int main()
{
    char* dbPath = "test.db";
    srand((unsigned int) time(nullptr));
    //srand(0x44556677);

    // Delete any existing database file
    unlink(dbPath);

    // Open the db
    unqlite* db = nullptr;
    if (unqlite_open(&db, dbPath, UNQLITE_OPEN_CREATE) != UNQLITE_OK)
    {
        printf("Error opening %s\n", dbPath);
        return 1;
    }

    // Insert data
    for (size_t i = 0; i < 200; i++)
    {
        // Create a key and record
        uint8_t key[40];
        uint8_t record[100];
        randomBytes(key, sizeof(key));
        randomBytes(record, sizeof(record));

        // Call unqlite_commit() each time since frequent commits
        // seem to be a cause of the problem
        if ((unqlite_kv_store(db, key, (int) sizeof(key),
                              record, sizeof(record)) != UNQLITE_OK) ||
            (unqlite_commit(db) != UNQLITE_OK))
        {
            printf("Error saving data\n");
            return 1;
        }
    }

    // Count how many records were inserted. The number should be correct.
    unqlite_kv_cursor* cursor = nullptr;
    if (unqlite_kv_cursor_init(db, &cursor) != UNQLITE_OK)
    {
        printf("Error creating cursor\n");
        unqlite_close(db);
        return 1;
    }

    size_t numInserted = 0;
    for (unqlite_kv_cursor_first_entry(cursor);
         unqlite_kv_cursor_valid_entry(cursor);
         unqlite_kv_cursor_next_entry(cursor))
    {
        numInserted++;
    }
    unqlite_kv_cursor_release(db, cursor);

    // Close and reopen the database
    unqlite_close(db);
    if (unqlite_open(&db, dbPath, UNQLITE_OPEN_CREATE) != UNQLITE_OK)
    {
        printf("Error reopening %s\n", dbPath);
        return 1;
    }

    // Count how many records are in file. The number
    // will be less than were inserted.
    cursor = nullptr;
    if (unqlite_kv_cursor_init(db, &cursor) != UNQLITE_OK)
    {
        printf("Error creating cursor\n");
        unqlite_close(db);
        return 1;
    }

    // This sometimes crashes if the db file is really messed up
    size_t numInFile = 0;
    for (unqlite_kv_cursor_first_entry(cursor);
         unqlite_kv_cursor_valid_entry(cursor);
         unqlite_kv_cursor_next_entry(cursor))
    {
        numInFile++;
    }
    unqlite_kv_cursor_release(db, cursor);
    unqlite_close(db);

    printf("Records inserted: %zu\n", numInserted);
    printf("Records in file: %zu\n", numInFile);

    return 0;
}

[symisc]

Thanks! That was a memory leak in unqlite_commit() that caused this data loss that we finally identified thanks to this report. Please update to UnQLite to 1.1.9 to fix this issue.

[kmvanbrunt]

Thanks a lot for the quick fix.