[tar] Progress callbacks

[infrafast]

I've tested setProgressCallback() with tarGzExpander and it works fine but look like it is not the case with tarExpander : nothing happens... I didn't debug much, maybe you can reproduce, if not I will perform additionnal investigation

[tobozo]

have you tried with setTarMessageCallback() ?

[infrafast]

Ok, I am nt sure to understand. The progress callback function we pass to tarGzExpander is pretty clear, it provide an int which indicate where we are so we can use it easily to display a bargraphprogress for instance

void myProgressCallback( uint8_t progress ) { Serial.printf("Progress: %d\n", progress ); }

However, setTarMessageCallback expect a function such the example you provided:

void myLogger(const char* format, ...)
{
  va_list args;
  va_start(args, format);
  vprintf(format, args);
  va_end(args);
}

I was expecting to have also a setProgressCallback that would work for TARexpander.. which for me as a different intent... would you mind showing me an example how to use myLogger to show progress of untaring , the code in myLogger is a bit obscure to me..

[tobozo]

oh right, tar isn't like gzip in a sense it doesn't hold the uncompressed global size: file sizes are accessible during unpacking but are relative to each target, not the source.

good news it it's eventually possible to attach a progress function to unTarStreamWriteCallback and unTarEndCallBack but it'll only be a incremented value, or a percentage of the file currently being unpacked.

another possibility when unpacking with an intermediate file (and the tar file size is known): unTarStreamReadCallback could eventually count the leftover bytes from the know tar file size.

I can research in that direction, but I'd be more prepared with a deeper knowledge on the situation of your .tar.gz file:

• is this .tar.gz file uploaded from the network or from a sketch data upload
• if using WiFi+WebServer, do you ESP.restart() before decompressing
• if using WiFi, what does ESP.getFreeHeap() return when the WiFi is turned off and before decompressing
• how many other files are in the tar.gz archive (size/path-depth)

[tobozo]

this branch implements setTarStatusProgressCallback and setTarProgressCallback for .tar.gz streams:

void myTarStatusProgressCallback( const char* name, size_t size, size_t total_unpacked )
{
  Serial.printf("[TAR] %-64s %8d bytes - %8d Total bytes\n", name, size, total_unpacked );
}

void myTarProgressCallback( uint8_t progress )
{
  static int8_t myLastProgress = -1;
  if( myLastProgress != progress ) {
    myLastProgress = progress;
    if( progress == 0 ) {
      Serial.print("Progress: [0% ");
    } else if( progress == 100 ) {
      Serial.println(" 100%]\n");
    } else {
      switch( progress ) {
        case 25: Serial.print(" 25% ");break;
        case 50: Serial.print(" 50% ");break;
        case 75: Serial.print(" 75% ");break;
        default: Serial.print("T"); break;
      }
    }
  }
}

setTarProgressCallback( myTarProgressCallback ); // prints the untarring progress for each individual file
setTarStatusProgressCallback( myTarStatusProgressCallback ); // print the filenames as they're expanded

[infrafast]

super cool! I'll test tomorrow. Here are the answer to your questions:

is this .tar.gz file uploaded from the network or from a sketch data upload : From network, it is basically used in the context where my ESP connects a webserver and check if there is a new version of firmware. This firmware includes binary and/or data files

if using WiFi+WebServer, do you ESP.restart() before decompressing: no, I call restart only if a .bin was part of the pack after it has been updated (after having called the update function of ESP). If the pack only contains data, I just uncompress them and continue the program.

if using WiFi, what does ESP.getFreeHeap() return when the WiFi is turned off and before decompressing : I I don't turn off wifi... should I ?

how many other files are in the tar.gz archive (size/path-depth): my tar is composed of: config1.conf - 359 byte index.html - 23182 byte logo.jpg - 5020 byte style.css - 5349 byte

I have to say your library bring new possibility: so far i found no method to be able to either update BIN or DATA or Both...I guess you are the only one providing such capability!!!

[tobozo]

yup WiFi eats some ram and tarGzExpander trades filesystem space vs 36KB of ram so it eliminates the use of the option without intermediate file.

it's not that bad to implement, just consider different states during setup() :

• state = update_available
• state = update_ready_to_deploy
• state = update_deployed

When state=update_ready_to_deploy:

• do no start WiFi
• attempt to decompress the archive
• change state to either update_available or update_deployed depending on the result of the decompression
• restart

With the help of system_get_rst_info(); it's even easy to spot between manual resets, software restart or crashes and avoid remotely bricking the ESP ^^

[tobozo]

I've "redesigned" and captured the new progress callbacks, this is as close to realtime as a GIF can be:

[infrafast]

thank you the callback works very well, did you integrate all in the main branch so you can updateb the library we download from Arduino IDE ?

[tobozo]

yup it's available in the Arduino IDE, but it breaks some wood and will require code change in existing projects so it's been tagged as alpha

[tobozo]

hey just a heads up before I close this issue, I'm preparing the next release based on some of your inputs.

This release introduces tarGzStreamUpdater(Stream*stream) as a mean to flash both the app and spiffs partitions from a .tar.gz stream (SD, Ethernet, CAN, HTTP, HTTPS or UDP) without using any intermediate file. The data partition can be SPIFFS or LITTLEFS

[infrafast]

Cool thanks for thé notification I’ll see to integrate it in my use case and keep you postes

Envoyé de mon iPhone

Le 15 févr. 2021 à 12:46, tobozo notifications@github.com<mailto:notifications@github.com> a écrit :

hey just a heads up before I close this issue, I'm preparing the next release based on some of your inputs.

This release introduces tarGzStreamUpdater(Stream*stream) as a mean to flash both the app and spiffs partitions from a .tar.gz stream (SD, Ethernet, CAN, HTTP, HTTPS or UDP) without using any intermediate file. The data partition can be SPIFFS or LITTLEFS

[image]https://user-images.githubusercontent.com/1893754/107935200-5d32b280-6f81-11eb-8c0f-1f1e6cfb36c5.png

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/tobozo/ESP32-targz/issues/20#issuecomment-779169106, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFAQXR6TDVZCSMW5WH6DWSDS7ECRXANCNFSM4WCMOEMA.