Problem with Sdfat library

[AlirezaAbdiyan]

Hi, I use SdFat library and use this code:

SdFat Sd;

#define tarGzFS Sd
#define FS_NAME "Sd"
#include <ESP32-targz.h>
...
TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();

    TARGZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARGZUnpacker->setTarVerify( true ); // true = enables health checks but slows down the overall process
    TARGZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    TARGZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    TARGZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity
    TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
    TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARGZUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback ); // tar log verbosity

    // using an intermediate file (default is /tmp/tmp.tar)
    if( !TARGZUnpacker->tarGzExpander(tarGzFS, "/Web upload files.tar.gz", tarGzFS, "/temp") ) {
      Serial.printf("tarGzExpander+intermediate file failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }

Error:
Compilation error: no matching function for call to 'TarGzUnpacker::tarGzExpander(SdFat&, const char [25], SdFat&, const char [6])'

Could you help me to solve it? Thanks.

[tobozo]

hi

ESP32-TarGz uses filesystem abstractions, but SdFat does not support it, so you'll need a custom implementation of fs::FS that wraps SdFat

#include "sd_fat32_fs_wrapper.h"
SdFat sd_fat_fs; 
fs::FS Sd = fs::FS(fs::FSImplPtr(new SdFat32FSImpl(sd_fat_fs)));

[AlirezaAbdiyan]

Thank you for your quick response. I followed the steps you mentioned, but I'm still getting error! In constructor 'SdFatFile32Impl::SdFatFile32Impl(FsFile)': C:\Users\hp\Desktop\Web upload files\SD_Update\sd_fat32_fs_wrapper.h:29:46: error: use of deleted function 'FsFile::FsFile(const FsFile&)'

error: 'FsBaseFile::FsBaseFile(const FsBaseFile&)' is private within this context

[tobozo]

well if you're using this version of SDFat then it's a SDFat problem

please see this issue, or just downgrade SDFat version

[AlirezaAbdiyan]

Thanks, I downgrade SDFat version and can compile it but program freeze !

My code:

#include <FS.h>
#include <LittleFS.h>
#include <SdFat.h>

const uint8_t SD_CS_PIN = 5;
#include <sd_fat32_fs_wrapper.h>

SdFat Sd;

fs::FS sd_fat_fs = fs::FS(fs::FSImplPtr(new SdFat32FSImpl(Sd)));

#define tarGzFS sd_fat_fs
#define FS_NAME "sd_fat_fs"
#include <ESP32-targz.h>

void setup() {
    Serial.begin(115200);
    pinMode(SD_CS_PIN,OUTPUT);
    if (!LittleFS.begin()) 
    {
       return;
    }

    if (!Sd.begin(SD_CONFIG)) {
        return;
    }
    else 
    {
    TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();

    TARGZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARGZUnpacker->setTarVerify( true  ); // true = enables health checks but slows down the overall process
    TARGZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    TARGZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    TARGZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity
    TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
    TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARGZUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback ); // tar log verbosity
    // using an intermediate file (default is /tmp/tmp.tar)
    /*if( !TARGZUnpacker->tarGzExpander(tarGzFS, "/Web upload files.tar.gz", tarGzFS, "/temp") ) {
      Serial.printf("tarGzExpander+intermediate file failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }*/

    // or without intermediate file
    if( !TARGZUnpacker->tarGzExpander(tarGzFS, "/1.tar.gz", tarGzFS, "/temp", nullptr ) ) {
      Serial.printf("tarGzExpander+intermediate file failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }
    Serial.printf("done!");

    }
}
void loop() {
}

Serial monitor: [TGZ] Will expand without intermediate file FileImplPtr open aufgerufen [TGZ] Will direct-expand /1.tar.gz to /temp

[tobozo]

how big is 1.tar.gz and how many files does it hold?

[AlirezaAbdiyan]

[Uploading 1.tar.gz…]() 842 kb and it contains two files. I change TARGZUnpacker->haltOnError( false ); to false and give this: tarGzExpander+intermediate file failed with return code #-2 And saw this error code is "Not a valid gzip file" but my file is correct!

[tobozo]

but my file is correct!

there are three ways to verify that:

• test the file with SD.h instead of SDFat.h
• attach the file to this thread so I can inspect it
• inspect it yourself with the following command line: gzip -t file.gz && echo "valid" || echo "NOT valid"

[edit] apparently you tried to attach it to this thread and it failed, so I'm tempted to say it's not valid

[AlirezaAbdiyan]

I appreciate your quick response. Thank you very much. I test my file with your command and this file is valid! 1.tar.gz

[AlirezaAbdiyan]

I want to compress the program file .bin and littlefs.bin of ESP32 along with some other information such as version, etc., and upload it to a web server so I can store it on a memory device. Then, I will decompress it and update the ESP32. Do you think this method is suitable, or do you suggest something else? The reason for using SdFat is that the program needs to continuously read a large amount of data from the memory, and as far as I understand, this library is better than SD. I would appreciate hearing your opinion on this.

[tobozo]

how big is the LittleFS partition? maybe it's full

until this is solved you should set the output debug level to debug in Arduino IDE, or add -DCORE_DEBUG_LEVEL=5 to build_flags in platformio.ini, this will slow down stuff but emit more debug messages

and as far as I understand, this library is better than SD

well SD.h has often been highlighted as unreliable so if your experience is similar then it's understandable to choose SdFat, however SdFat is untested with ESP32-tarGz and additional tweaks may be needed to get it working

I want to compress the program file .bin and littlefs.bin of ESP32 along with some other information such as version, etc.

from what I understand, you want to do some OTA updates using firmware+filesystem compressed in tgz format? if so TarGzUnpacker::tarGzStreamUpdater() may be more appropriate.

if you prefer to flash firmware and filesystem separately (as in a transaction) there are two other OTA-specialized functions that work without tar (gzip only): GzUnpacker::gzUpdater() and GzUnpacker::gzStreamUpdater()

you can even drop the fs::FS layer for SdFat and just stick with the *Stream* methods (i.e. tarGzStreamExpander(), tarGzStreamUpdater(), gzStreamUpdater())

if firmware version control is really important for your project, you should take a look at esp32FOTA which I have extented to use GzUpdateClass from esp32-targz, it suggests a workflow using a json manifest file where you can specify a firmware version, and it handles gzipped files transparently, but it comes with a lot of additional complexity and a steep learning curve

[AlirezaAbdiyan]

Thank you for your suggestion . I will use the update functions of your library and hope I won't encounter any issues.

[AlirezaAbdiyan]

I downgraded the SdFat library and used your library, and now my program is working correctly. Thank you for your excellent library and guidance.