Decompress from byte array in PROGMEM to FS

[bakadave]

I have a byte array in PROGMEM (compressed html) that I want to decompres to SPIFFS. What I'm currently doing is copying the byte array in to index.html.gz and uncompressing it from the SPIFFS to index.html. The problem is that this means at one point the data exists 2x in SPIFFS (and 3x overall), and this project is incredibly flash-constrained.

Ideally I would like to decompress straight from a byte array to SPIFFS. Is it possible with this library?

[tobozo]

hi,

you can use a ReadBufferStream to create a stream from progmem, and pass that stream to esp32-targz

however:

• gz files served by webserver libraries probably don't need to be decompressed, most clients understand Accept-Encoding: gzip headers, and most webserver libraries for ESP32 handle that natively
  • SPIFFS for ESP32 is deprecated, LittleFS is the replacement; it is more reliable, suffers less data corruption/performances problems and doesn't wear out your flash prematurely

[bakadave]

you can use a ReadBufferStream to create a stream from progmem, and pass that stream to esp32-targz

thank you I'll to do that

gz files served by webserver libraries probably don't need to be decompressed, most clients understand Accept-Encoding: gzip headers, and most webserver libraries for ESP32 handle that natively

you are 100% right by I need a template processor to process it and I need to decompress for that

SPIFFS for ESP32 is deprecated, LittleFS is the replacement; it is more reliable, suffers less data corruption/performances problems and doesn't wear out your flash prematurely

right again, but I'm pressed on flash and SPIFFS has a much lower overhead than LittleFS

[bakadave]

related to PR #74

[tobozo]

thanks a lot for your contribution 👍 I've created a new release 1.2.3 which should propagate soon in Arduino registry, it may take some time with platformio registry though

I'll add an example based on your use case (progmem to flash filesystem) and update the readme later next week when I can get my hands on a pico and 8266.

[bakadave]

Good point regarding the example. Shall I push my implementation to the PR?

[tobozo]

appreciated but no thanks: the function example doesn't need a webserver and should be usable in unit tests and QEMU;

also it's too early to decide where to put that example as I need to compare it with the counterpart implementation

I haven't tested yet but I believe the same goal could be achieved using setStreamWriter()

// pseudo code, may contain syntax/compile errors

static File myOutputFile;

bool myStreamWriter( unsigned char* buff, size_t buffsize )
{
  return myOutputFile.write( buff, buffsize ) > 0;
}

void gzStreamExpandIndexHTML() 
{
  myOutputFile = SPIFFS.open("/index.html", "w");
  if( !myOutputFile ) return; 

  GzUnpacker *GZUnpacker = new GzUnpacker();

  GZUnpacker->setStreamWriter( myStreamWriter );

  ReadBufferStream stream(index_html_gz, index_html_gz_len); 

  if( !GZUnpacker->gzStreamExpander( &stream, index_html_gz_len ) ) {
    Serial.printf("gzStreamExpandIndexHTML failed with return code #%d", GZUnpacker->tarGzGetError() );
    Serial.println( );
  } else {
    ret = true;
  }

  myOutputFile.close();
}

if the performances are similar then some refactoring will happen