Using ESP32-targz notes for article

[frankcohen]

HI Tobozo, I love your ESP32-targz library. I am using it in my wrist watch project: https://github.com/frankcohen/ReflectionsOS. I spent around 20 hours getting it to work with my ESP32 and SD card set-up. I bumped into a bunch of problems of my own making, and some that the library just didn't handle. Below are my notes. I intend to write these up as a Reddit post to share my experience and recommend ESP32-targz. I would appreciate any feedback/edits. Thanks! -Frank

Using TAR.GZ files in ESP32 applications

Why use TAR.GZ files in a communication sketch: Archives decompress multiple files at once: less chance for errors during transmission Archives guarantee their file contents are the same after transfer Archives decompress into a file/directory structure on the SD card Zip decompression libraries require more memory and are more complicated to code

Using ESP32-targz library to uncompress tar files https://github.com/tobozo/ESP32-targz

Library requires this to be defined ahead of the include, it only works this way

#define DEST_FS_USES_SD
#include <ESP32-targz.h>

boolean tarDecompress()
{
  bool ret = false;
  const char* tarGzFile = "/startup.tar.gz";
  const char * mytmpdest = "/tempo"; // for md5 tests

  Serial.println( "Expanding archive" );

  TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();

  TARGZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
  TARGZUnpacker->setTarVerify( true ); // true = enables health checks but slows down the overall process

  TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
  TARGZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
  TARGZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
  TARGZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity
  TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
  TARGZUnpacker->setTarMessageCallback( myTarMessageCallback ); // tar log verbosity

  Serial.println("Testing tarGzExpander without intermediate file");
  if( !TARGZUnpacker->tarGzExpander(tarGzFS, tarGzFile, tarGzFS, mytmpdest, nullptr ) ) {
    Serial.print("tarGzExpander direct expanding failed with return code: ");
    Serial.println( TARGZUnpacker->tarGzGetError() );
  } else {
    ret = true;
  }

  if ( ret )
  {
    Serial.println( "Expanded successfully" );
  }
  else
  {
    Serial.println( "Expand failed" );
  }
  return ret;        
}

Things I got wrong/Mistakes I made:

Warnings about 8.3 file names, using ESP32, standard SD library, works fine with longer file and directory names

File extensions confuse the library. For example, I included startup.ref. Library stopped decompressing on startup.ref with no error. Solved by using .txt file extension.

MacOS special handling

MacOS files confuse the library. Library stopped decompressing at ./DS_Store, .Spotlight-V100 stopped decompressing with no error. Solution is to use the undocumented flag when creating the tar:

#! /bin/bash
cd /Users/frankcohen/Desktop/ReflectionsExperiments
rm startup.tar.gz
COPYFILE_DISABLE=1 tar czf startup.tar.gz --exclude ".DS_Store" startup

MacOS comes with bsdtar installed, and symlink-ed to tar.

[frankcohen]

I'm finding that the library does not extract a text file. It extracts binary files (like .m4a audio and .mpg video files) and simply ignore .txt files and files with a .ref (my own extension) containing JSON encoded values.

[tobozo]

Hi Frank,

thanks for documenting this library and spotting that issue, I'll close this and continue the discussion on the other thread

[tobozo]

I've edited your first comment, thanks again for your effort.