infrafast commented 3 years ago

Hello thanks for sharing this great library! when untaring a file I received the following error: [TAR ERROR] File size and data size do not match (545008 vs 545008)! see details below... Seems that either the condition of testing this error may have an isse since it indicate the same size.... Would you have alook to this , if needed I can provide more details FYI I managed to untar a tar that contained smaller files succesuflly,

starting dezip process[TAR DEBUG]: Copying tar header

  filename: 2.32.ino.d1_mini.bin
  filemode: 0100777 (33279)
       uid: 00 (0)
       gid: 00 (0)
  filesize: 02050360 (545008)
     mtime: 013777612447 (1610552615)
  checksum: 011225 (4757)
      type: 0

link_target: ustar ind: ustar ustar ver: 00 user name: group name: device (major): 0 device (minor): 0 data blocks = 1065 last block portion = 240

[TAR] Creating /tmp/2.32.ino.d1_mini.bin [TAR ERROR] File size and data size do not match (545008 vs 545008)!

tobozo commented 3 years ago

thanks for your feedback

if both sizes match in the error message it means the decompression was successful but the file that should have been created from this data does not exist on the filesystem, maybe you hit a partition full while decompressing?

If you call this before decompression you will probably get a more relevant error message:

setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn );

There are a few things you can try:

Set Core Debug Level to Debug
use tarGzExpander without intermediate file (see the ReadMe or the example for the syntax)
format your SPIFFS partition and try again (SPIFFS is easily corrupted)
use setTarVerify( false ) before unpacking to bypass the health checks (this won't prevent the partition from getting full though)

infrafast commented 3 years ago

Thanks for your fast feedback, here are my inputs

Actually my code already uses setupFSCallBacks () and I use LittleFS, not SPIFFS, see an extract below.

        LittleFS.remove((upload.filename+".BAK").c_str());
        //setProgressCallback( unpackProgressCallback );
        if (filename.endsWith(".tar")){
          DEBUG_PRINT("\nstarting dezip process");  
          **setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn );**
          if( ! tarExpander(LittleFS, filename.c_str(), LittleFS, "/tmp") ) {
            DEBUG_PRINT("operation failed with return code #%d", tarGzGetError() );
          }
          LittleFS.remove(filename);
        }else if (filename.endsWith(".tgz")){
          DEBUG_PRINT("\nstarting dezip process");  
          **setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn );**
          if( ! tarGzExpander(LittleFS, filename.c_str(), LittleFS, "/tmp","/tmp/unpack.tmp") ) {
            DEBUG_PRINT("operation failed with return code #%d", tarGzGetError() );
          }
          LittleFS.remove(filename);
        }

CASE 1 IF WE HAVE A TAR FILE TO UNTAR:

For the time being, my concern is that when using UNTAR function, the output of the log is the following:

------------------------------
File system info
------------------------------
**Total space:      14655488 byte
Total space used: 663552 byte**
Block size:       8192 byte
Page size:        14655488 byte
Max open files:   5
Max path lenght:  32

------------------------------
List files
------------------------------
config1.conf - 359 byte
index.html - 23182 byte
logo.jpg - 5020 byte
style.css - 5349 byte
tmp - this is a folder
------------------------------

Upload finished

starting dezip process[TAR DEBUG]: Copying tar header
===========================================
      filename: 2.33.ino.d1_mini.bin
      filemode: 0666 (438)
           uid: 00 (0)
           gid: 00 (0)
      filesize: 02150060 (577584)
         mtime: 014000026000 (1610624000)
      checksum: 012374 (5372)
          type: 0
   link_target: 
     ustar ind: ustar
     ustar ver: 00
     user name: 
    group name: 
device (major): 0
device (minor): 0
  data blocks = 1129
  last block portion = 48
===========================================
[TAR] Creating /tmp/2.33.ino.d1_mini.bin
[TAR ERROR] File size and data size do not match (**577584 vs 577584**)!
[TAR ERROR]: End callback failed.
[ERROR] operation aborted while expanding tar file 2.33.tar (return code #-34
operation failed with return code #-34

As you can see, there is really enough space on the file system; The decompress file is also present on the file system:

/tmp/2.33.ino.d1_mini.bin   qnd has a size of **577 584** 

CASE 2 : IF WE HAVE A TGZ to UNCOMPRESS

No matter if I use or not intermediate file for the TGZ, it will fail. 

tarting dezip processInsufficient heap to decompress (available:11392, needed:36864), aborting
uzLib expander start!
[GZ INFO] valid gzip file detected! gz size: 410068 bytes, expanded size:609280 bytes
[GZ INFO] Available space:14155776 bytes
[INFO] gzUncompress tradeoff: faster, used 36864 bytes of ram (heap after alloc: 10792)
[INFO] output_buffer_size=4096 blockmod=8
[GZ WARNING] uzlib_uncompress_chksum had a premature end: -4 at position 3072 while 1076013364 bytes left
decompressed 609280 bytes
uzLib expander finished!
[GZ Info] FreeBytes after expansion=13541376
[INFO] heap before tar-expanding: 10672)
[TAR DEBUG]: Copying tar header
===========================================
      filename: 2.33.ino.d1_mini.bin
      filemode: 0666 (438)
           uid: 00 (0)
           gid: 00 (0)
      filesize: 02134140 (571488)
         mtime: 014000002650 (1610614184)
      checksum: 012402 (5378)
          type: 0
   link_target: 
     ustar ind: ustar
     ustar ver: 00
     user name: 
    group name: 
device (major): 0
device (minor): 0
  data blocks = 1117
  last block portion = 96
===========================================
[TAR] Creating /tmp/2.33.ino.d1_mini.bin
[TAR DEBUG]: Copying tar header
===========================================
      filename: .4280.141 SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
      filemode: 00 (0)
           uid: 00 (0)
           gid: 00 (0)
      filesize: 00 (0)
         mtime: 013777775065 (1610611253)
      checksum: 011260 (4784)
          type: 0
   link_target: 
     ustar ind: uzԈ⸮
     ustar ver: 00
     user name: 
    group name: 
device (major): 0
device (minor): 0
  data blocks = 0
  last block portion = 512
===========================================
[ERROR] in unTarHeaderCallBack: Could not open [/tmp/.4280.141 SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS] for write.
[TAR ERROR]: An error occured during Header callback.
[ERROR] operation aborted while expanding tar file /tmp/unpack.tmp (return code #-33
operation failed with return code #-33

Note: with setTarVerify( false );

the TGZ will not generate any error for the first file but it will look for a secnd file name tmp/.4280.141 SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS ...

The TAR will justwork fine !

tobozo commented 3 years ago

I edited your comment for readability :-)

With no intermediate file tarGzExpander throws an error Insufficient heap to decompress (available:11392, needed:36864), aborting but doesn't return, this has been fixed in the latest release.

The problem is that without intermediate file, a gzip dictionnary is required (36Kb ram), and I'm not sure there are many situations where these conditions are met.

tobozo commented 3 years ago

au fait on peut parler français ^^

tobozo commented 3 years ago

Explication possible pour le message [TAR ERROR] File size and data size do not match (**577584 vs 577584**)! :

      if( pos != proper->filesize || !tarFS->exists(tmp_path ) ) {
        log_e("[TAR ERROR] File size and data size do not match (%d vs %d)!", pos, proper->filesize);
        delete tmp_path;
        return ESP32_TARGZ_FS_WRITE_ERROR;
      }

Comme on voit dans le message que pos est égal a proper->filesize on peut déduire que c'est le second test tarFS->exists(tmp_path ) qui a échoué (on peut également déduire que ce message d'erreur a été fait a l'arrache :roll_eyes:).

Vu que le fichier est bien la apres ce message, on peut spéculer sur le fait que le nom du fichier est trop long ( /littlefs/tmp/2.33.ino.d1_mini.bin = 35 caracteres ) ce qui ferait échouer le test d'existence du fichier car SPIFFS autant que LittleFS ont une limitation sur la longueur des chemins générés (32 caracteres)

Pour vérifier cette these, il suffit donc de réduire le nom du fichier a quelque chose de plus court (ex: /littlefs/tmp/2.33.d1_mini.bin = 31 caracteres).

Si c'est vérifié je devrai alors tenir compte du préfixe posix.

infrafast commented 3 years ago

Super oui :-) Merci pour la réactivité et le support. Here is the outome of my further test.

indeed it is due to the filename too long: I changed it to 2.34.bin and remove the //setTarVerify( false ); and now working well!

Brilliant, you could maybe adjust your code to provide a clearer error message and close the issue

tobozo commented 3 years ago

should be fixed by the latest release, feel free to reopen if the problem comes back, closing this

tobozo / ESP32-targz

File size and data size do not match (545008 vs 545008)! #19

starting dezip process[TAR DEBUG]: Copying tar header

link_target: ustar ind: ustar ustar ver: 00 user name: group name: device (major): 0 device (minor): 0 data blocks = 1065 last block portion = 240