tar works but tar.gz return -33 error

[infrafast]

Hello Tobozo, first of all thanks again for your lib. this is really working great! I face one issue , when using a tar it's working good, but when using a tar.gz, I got a -33 error (Error TAR_ERR_HEADERCB_FAIL). here is the code (quite standard)

  TARGZUnpacker.setTarProgressCallback( unpackProgressCallback ); 
  TARGZUnpacker.setTarStatusProgressCallback(tarStatusProgressCallback ); 
  TARGZUnpacker.setGzProgressCallback( unpackProgressCallback);
  if (fileName.endsWith(".tar") || fileName.endsWith(".tar.gz")) {
    if ( ! TARGZUnpacker.tarExpander(LittleFS, fileName.c_str(), LittleFS, "/") ) {
      success = false;
      // TARGZUnpacker.tarGzGetError());  ==> return -33
    }else{
       success = true;
    }
  } 

My assumption is that my tar.gz file has a format issue ?

Here are the command I use to generate my file:

tar -C src\data -cvf archive.tar *.*
tar -C src\data -zcvf archive.tar.gz *.*

the files link if it may help: (copy paste in another windows as this is not https and github will reject the link) http://infrafast.com/archive.tar.gz http://infrafast.com/archive.tar thanks a lot!

ps: I am using lib version 1.1.4

[tobozo]

hi, thanks for your feedback :+1:

Here's how I generate tar.gz files from the command line:

tar -cvzf data.tar.gz src/data

But it also pulls the src/data folder path in the archive, and it's probably not what you want.

If doing tar and gz separately, only use tar in the first step, and gzip in the second step:

tar -C src\data -cvf archive.tar .
gzip archive.tar

If this does not solve the issue, I'll need you to provide extra info such as device (ESP32/ESP8266/RP2040), available heap, platform and core version , and a faulty .tar.gz file so I can reproduce the problem locally.

BTW the current development version of ESP32-Targz is ahead of the release version: https://github.com/tobozo/ESP32-targz/tree/1.1.8 so if you feel like experimenting you may also try this one.

[infrafast]

tar -cvzf data.tar.gz src/data But it also pulls the src/data folder path in the archive, and it's probably not what you want.

yep, that's why I use the -C option first

If doing tar and gz separately, only use tar in the first step, and gzip in the second step:

Ok, I've tried as you suggested but it doesnt work... the file I generate in two tsep is http://infrafast.com/archive-2steps.tar.gz

If this does not solve the issue, I'll need you to provide extra info such as device (ESP32/ESP8266/RP2040), available heap, platform and core version, and a faulty .tar.gz file so I can reproduce the problem locally.

_Here are the info:

CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/d1_mini.html PLATFORM: Espressif 8266 (4.1.0) > WeMos D1 R2 and mini HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash PACKAGES:

• framework-arduinoespressif8266 @ 3.30101.0 (3.1.1)
• tool-esptool @ 1.413.0 (4.13)
• tool-esptoolpy @ 1.30000.201119 (3.0.0)
• tool-mklittlefs @ 1.203.210628 (2.3)
• tool-mkspiffs @ 1.200.0 (2.0)
• toolchain-xtensa @ 2.100300.220621 (10.3.0)_

runtime debug info: free heap size before calling unpacker: 6112 (I know it's not much, when I disabled all debug trace, etc, I get ~10K of heap and still have the issue)

BTW the current development version of ESP32-Targz is ahead of the release version: https://github.com/tobozo/ESP32-targz/tree/1.1.8 so if you feel like experimenting you may also try this one.

well, in my platformIO.ini file I have tobozo/ESP32-targz which uses the latest (1.1.7), I tried to add tobozo/ESP32-targz@1.1.8 but doesnt find it, by chance could you let me know which line I should use to go with 1.1.8 ?

[tobozo]

I'm not sure how to tweak platformio to pull a git branch, but I still don't know what device to test with, is that an esp32, esp8266 or rp2040 ?

I managed to decompress your archive.tar.gz on an ESP32 in verify mode, so the file format/contents is not the cause.

Decompression Logs

```log 11:44:48.060 -> ├──────────────────────────────────────────────────────────────────┤ 11:44:48.094 -> | Testing tarGzExpander with intermediate file | 11:44:48.094 -> └──────────────────────────────────────────────────────────────────┘ 11:44:48.094 -> [TGZ] Will expand using an intermediate file: /tmp/data.tar 11:44:48.127 -> [GZ] Expanding /archive.tar.gz to /tmp/data.tar 11:44:49.188 -> Progress: 0% ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 25% ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 50% ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 75% ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 100% 11:45:11.924 -> [TAR] /tmp/data.tar can't be checked 11:45:12.023 -> [TAR] Expanding stream to folder /tmp 11:45:13.183 -> [TAR] cable.jpeg 1173 bytes - 1173 Total bytes 11:45:13.183 -> Progress: 0% ▓▓▓▓ 100% 11:45:13.415 -> [TAR] cable.jpeg MD5 check SUCCESS! 11:45:14.640 -> [TAR] cable.jpg 1432 bytes - 2605 Total bytes 11:45:14.640 -> Progress: 0% ▓▓▓▓ 100% 11:45:14.872 -> [TAR] cable.jpg MD5 check SUCCESS! 11:45:16.132 -> [TAR] common.min.js.gz 3088 bytes - 4520 Total bytes 11:45:16.132 -> Progress: 0% ▓▓▓▓▓▓▓▓ 100% 11:45:16.397 -> [TAR] common.min.js.gz MD5 check SUCCESS! 11:45:18.453 -> [TAR] CONFIG.jpeg 2466 bytes - 5554 Total bytes 11:45:18.453 -> Progress: 0% ▓▓▓▓▓▓ 100% 11:45:18.917 -> [TAR] CONFIG.jpeg MD5 check SUCCESS! 11:45:19.846 -> [TAR] CONFIRM.jpeg 2660 bytes - 5126 Total bytes 11:45:19.879 -> Progress: 0% ▓▓▓▓▓▓▓ 100% 11:45:20.078 -> [TAR] CONFIRM.jpeg MD5 check SUCCESS! 11:45:21.072 -> [TAR] customstepsjs.min-1.0.0.js.gz 1407 bytes - 4067 Total bytes 11:45:21.105 -> Progress: 0% ▓▓▓▓ 100% 11:45:21.304 -> [TAR] customstepsjs.min-1.0.0.js.gz MD5 check SUCCESS! 11:45:22.364 -> [TAR] deutsch.jpeg 1374 bytes - 2781 Total bytes 11:45:22.397 -> Progress: 0% ▓▓▓▓ 100% 11:45:22.596 -> [TAR] deutsch.jpeg MD5 check SUCCESS! 11:45:23.690 -> [TAR] deutsch.json.gz 4390 bytes - 5764 Total bytes 11:45:23.723 -> Progress: 0% ▓▓▓▓▓▓▓▓▓▓ 100% 11:45:24.021 -> [TAR] deutsch.json.gz MD5 check SUCCESS! 11:45:25.148 -> [TAR] deutsch.locale 823 bytes - 5213 Total bytes 11:45:25.181 -> Progress: 0% ▓▓▓ 100% 11:45:25.414 -> [TAR] deutsch.locale MD5 check SUCCESS! 11:45:26.607 -> [TAR] DOWNLOAD.jpeg 1585 bytes - 2408 Total bytes ```

I misread your initial post, at first I thought you were pasting a positive result, but you're using the same code block both for .tar and .tar.gz files, right? If so, then this is the cause.

Although using tarExpander() is fine for .tar files, it should be tarGzExpander() when using .tar.gz files.

[infrafast]

I'm not sure how to tweak platformio to pull a git branch, but I still don't know what device to test with, is that an esp32, esp8266 or rp2040 ?

sorry if it was not clear, the device to test was indicated into my previous post: PLATFORM: Espressif 8266 (4.1.0) > WeMos D1 R2 and mini HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash

I misread your initial post, at first I thought you were pasting a positive result, but you're using the same code block both for .tar and .tar.gz files, right? If so, then this is the cause.

Although using tarExpander() is fine for .tar files, it should be tarGzExpander() when using .tar.gz files.

Oh my god, how could I miss that.... I've used tarGzExpander. I've tested with it, When sGzexpander is calledl looks like the esp freezem then watchdog is triggered

starting expansion of: archive.fsaz ets Jan 8 2013,rst cause:4, boot mode:(3,7)

wdt reset load 0x4010f000, len 3424, room 16 tail 0 chksum 0x2e load 0x3fff20b8, len 40, room 8 tail 0 chksum 0x2b csum 0x2b v000940f0 ~ld

I will check if this is due to memeory or FS fill or whatever and reports here if I can not find why... unless you have an obvious idea.

[tobozo]

tar.gz expanding can be memory hungry indeed, the latest branch has some tweaks for that and also for the wdt reset: ther is a missing yield() in the tar decompression loop, SPIFFS and SD don't care if it's missing but LittleFS needs it.

Here's how you can use the current development/unstable branch in your platformio.ini

lib_deps =
  https://github.com/tobozo/ESP32-targz#1.1.8

[infrafast]

thanks for the hint. I have now switched to 1.1.8.

Interestingly, when I do not use the log callback function TARGZUnpacker.setLoggerCallback(BaseUnpacker::targzPrintLoggerCallback); => system watchdog is triggered

when I activate log, the code works but I get the following debug message

starting tgz expansion of: test.fsaz[TGZ] Will expand using an intermediate file: /tmp/data.tar [GZ] Expanding test.fsaz to /tmp/data.tar [E][ESP32-targz-lib.cpp:1278] gzUncompress(): [ERROR] can't alloc 1024 bytes for output buffer [E][ESP32-targz-lib.cpp:1465] gzExpander(): gzUncompress returned error code -106

I managed to free the required memory (without debug) and it works successuflly.

The only problem I thought this would already be usefull to know that wtch raise if you have no debug log output activated, this should probably be fixed for productive use.

[tobozo]

I could make the gz dictionary disabled by default on esp8266. It would prevent the allocation of 32Kb of heap, and disable dictionary based decompression. An intermediary file would become mandatory for .tar.gz files though

Plus crc based decompression is much slower and the spiffs partition may explode in the process, if the size becomes an issue, using gzStreamUpdater() on U_SPIFFS with an incoming HTTP stream may be a safer solution.

Reminder: ESP8266 has built-in uzlib so you may transparently use firmware.gz or spiffs.gz directly with the Update class

[tobozo]

if you delete your .pio folder and re-pull the 1.1.8 branch, there's a new GzUnpacker->noDict() method you can now call before decompression, this sould solve the memory problem, and even let you enable debug.

    TARGZUnpacker.noDict();
    TARGZUnpacker.tarGzExpander(/*...*/);

or if you want to try dictionary-based first and use crc-based on failure:

if ( ! TARGZUnpacker.tarGzExpander(/*...*/) ) {

  if ( TARGZUnpacker.getError() == ESP32_TARGZ_HEAP_TOO_LOW ) {

    TARGZUnpacker.noDict();
    TARGZUnpacker.tarGzExpander(/*...*/);

  }

}

[infrafast]

Thanks a lot for all information, I've tested 1.1.8 with the code update and this works well :-) you can close the issue