Fixes and Improvements

[savejeff]

Hi, as discussed in the other thread, here are the improvements I implemented for the lib

55

I have attached multiple versions that are more and more modified from the original version. the earlier might be easier to integrate.

sqlite3lib_github_fixes.zip

i renamed esp32.cpp with jhal.cpp

V1: only minor changes but added debug prints for understanding what the lib is doing V2: most of the fixes implemented but still similar to the original code <- I recommend this for importing fixes V3: refactoring and cleanup V4: my current version with working truncate.

V2-V4 uses my own implementation of a filesystem. it has the same functions as the esp fs classes like sd, sd_mmc, spiffs. replacing with the esp functions should be very easy.

if you have a question, just contact me. I recommend using a diff tool like beyond compare to compare the changes

Here are some additional explanations:

################## The File Struct #######################

struct ESP32File {
    sqlite3_file base;              /* Base class. Must be first. */
    jFileSystemClass* jfs;  // pointer to filesystem (replace this with esps fs instance like sd, mmc, spiffs, ...)
    jFile* fp;                      // pointer to file (replace this with File from esps fs::File 
    int lock;               // this is the lock applied to this database file
    int id;             // unique identifier for the database file to make debug prints easy

    char *aBuffer;                  /* Pointer to malloc'd buffer */
    int nBuffer;                    /* Valid bytes of data in zBuffer */
    sqlite3_int64 iBufferOfst;      /* Offset in file of zBuffer[0] */
};

##################### Log functions #########################

Log and LogD are just two levels of logging where LogD indicates debug prints only to the console, Log could also log to a logfile. just replace all LogD with Log

#if JSQL_DEBUG==1 // <- set JSQL_DEBUG to 1 to enable debug printing
#define LogX Log
#define LogXD LogD // <- remove "D" to only use Log
#else
#define LogX(...)
#define LogXD(...)
#endif

#include <stdarg.h>

String s_printf(const char* format, ...)
{
    char loc_buf[64];
    char* temp = loc_buf;
    va_list arg;
    va_list copy;
    va_start(arg, format);
    va_copy(copy, arg);
    int len = vsnprintf(temp, sizeof(loc_buf), format, copy);
    va_end(copy);
    if (len < 0) {
        va_end(arg);
        return "";
    };
    if (len >= sizeof(loc_buf)) {
        temp = (char*)malloc(len + 1);
        if (temp == NULL) {
            va_end(arg);
            return "";
        }
        len = vsnprintf(temp, len + 1, format, arg);
    }
    va_end(arg);

    String s = String(temp);

    if (temp != loc_buf) {
        free(temp);
    }
    return s;
}

extern void Log(const char* format, ...) __attribute__((format(printf, 1, 2)));
void Log(const char* format, ...)
{
    char loc_buf[64];
    char* temp = loc_buf;
    va_list arg;
    va_list copy;
    va_start(arg, format);
    va_copy(copy, arg);
    int len = vsnprintf(temp, sizeof(loc_buf), format, copy);
    va_end(copy);
    if (len < 0) {
        va_end(arg);
        return;
    };
    if (len >= sizeof(loc_buf)) {
        temp = (char*)malloc(len + 1);
        if (temp == NULL) {
            va_end(arg);
            return;
        }
        len = vsnprintf(temp, len + 1, format, arg);
    }
    va_end(arg);

    SerialOut.print("[");
    SerialOut.print(millis());
    SerialOut.print("] ");
    SerialOut.println(temp);

    if (temp != loc_buf) {
        free(temp);
    }
}

void Log(const String& msg)
{
    Log(msg.c_str());
}

##################### truncate #############################

here the implementation of truncate in my file system class:

here the file flags:

// Read and Write with Position at end of file. appends starting from end of file
#define _FILE_FLAG_RW_APPEND "a+"

// Read and Write with Position at start file. overrides starting from position 0
// Can be replaced by APPEND and move to first position (seek(0))
#define _FILE_FLAG_RW "r+"
#define _FILE_FLAG_RW_OVERRIDE "w+"

// For read file only
// in General this might work better than w+ when reading a whole file from start to end
#define _FILE_FLAG_R "r"

bool jFileSystemClass::truncate(const String& path, uint32_t size) 
{
    if(!exists(path))
        return false;

    String ftmp = get_tmp_filename();

    LogXD("jfs.%s.truncate: ftmp=%s", get_tag(), ftmp.c_str());

    rename(path, ftmp);

    jFile* f_old = open(ftmp, jFILE_MODE::FILE_FLAG_R);
    jFile* f_new = open(path, jFILE_MODE::FILE_FLAG_RW_OVERRIDE);

    //if(!f_old->isOpen() || !f_new->isOpen())
    //  return false;

    uint32_t size_new = 0;
    uint8_t buff[128];
    while (size_new < size && f_old->available())
    {
        int len = MIN(128, size - size_new);
        int rlen = f_old->read(buff, len);

        f_new->write(buff, rlen);
        size_new += rlen;
    }

    LogXD("jfs.%s.truncate: size=%d, size_new=%d", get_tag(), size, size_new);

    f_old->close();
    f_new->close();

    delete(f_old);
    delete(f_new);

    remove(ftmp);

    return true;
}

String jFileSystemClass::get_tmp_filename() 
{
    //TODO deadlock in case of all temp files are present -> not likely
    while(true) {
        int x = get_random_int(0, 999);
        String fname = s_printf("TMP%02d.BIN", x);
        if(!exists(fname))
            return fname;
    }
}

[siara-cc]

Hi @savejeff , Thanks for sharing this and sorry about the late response. I will work on it and update the library soon!!

[savejeff]

no problem. just wanted to give something back to the open source community.

I think making SQlite available to the Arduino community is a big thing. With the Filesystem abstraction, I'm also able to use the SdFat library and target other processors like RP2040 and SAMD etc. Teensy still has some problems though