allow 70gb+ files using a 4 byte marker and then 64 bit binary in nbo

[monkeyiq]

I had a user report failure for large single files (70gb+) created using ArchiveStream. From digging into things gnu tar is happy to make such archives and will use a marker 0x80 0x0 0x0 0x0 in nbo then the 64 bit size in binary stored in nbo in the 12 byte size field.

I have only tested this patch on a few files, one with all zeros and then one which appends the Fedora linux image to itself a number of times to make a single 84gb file.

dd if=Fedora-KDE-Live-x86_64-41-1.4.iso of=bigtestfile bs=1G oflag=append conv=notrunc
md5sum  bigtestfile 
a1f28fa449d328ef410e207ad15040b3  bigtestfile

I made a very basic test script to make a tar archive from a directory which made a tar file that gnu tar was happy to view and extract

$ mkdir -p testdir
$ mv bigtestfile testdir
$ php tar-test.php made-with-filesender.tar testdir
$ mkdir out
$ cd out
$ tar tvf ../made-with-filesender.tar 
-rwxrwxrwx 0/0     89804242944 2024-11-06 09:22 bigtestfile
$ tar xvf ../made-with-filesender.tar 
bigtestfile
$ md5sum  bigtestfile 
a1f28fa449d328ef410e207ad15040b3  bigtestfile

So this appears to work ok. I am of the mind that if gnu tar can extract it ok then that is the pillar that I base "works ok" on. At least for initial testing. Obviously it would be handy to track down information on if something should be changed in addition to this to indicate that the size is in binary. Also seeing if other extractors are happy to touch these generated tar files.