Flexible "pipe" compression mechanism for user compression schemes

(Note: this is a feature request, not an issue per se.)

According to the "Why xar" document:

"Files in xar are individually compressed. This allows for quick extraction
of individual files without the extra disk space requirements and CPU usage
of extracting the entire archive, as compared to a compressed tar archive.
This makes xar useful for quick restores of accidentally deleted or
overwritten files, from a backup archive."

I was looking at an efficient random-access format exactly for this backup
scenario.  I am looking at compressing and encrypting files with GPG when
storing them in the archive.  Right now, using tar, it means tar-ing it
all, than gpg-ing the whole archive.  So to get a file from the archive,
you need to un-gpg-it globally, then use tar to extract the file.  With xar
too, it would mean un-gpg-ing it globally before seeing the contents.

I was thus thinking it could be great to have something like
--compression=pipe and then a --compress-pipe=<command>.  The command would
be fed with the input file and output the compressed file, then stored in
the archive.

In my scenario, it would make it possible to do something like

xar ... --compress=pipe --compress-pipe='gpg -c --output -'

However, problems arise at decompression:

a) User needs to know how it was compressed
   Possible solution: store the 'compress-pipe=<...>' that was used in 
   the xar metadata to give a clue to the user

b) User needs to specify how to uncompress
   Again, at uncompress, use something like
   --compress=pipe --uncompress-pipe='gpg -d -'

   At compression, there could also be an optional 
   --suggested-uncompress-pipe=<> that would store metadata so that the 
   receiving end has a clue.  (But should never be executed at 
   decompression, as it can be a major security flaw...)

Having such a mechanism (most probably a little more refined) would make
xar a perfect choice for random access archives where file are stored
according to user specifications.

I hope these ideas can fuel some discussion.  Thanks.

Original issue reported on code.google.com by jf.rich...@gmail.com on 20 Nov 2007 at 9:48

jmichaelh / xar

Flexible "pipe" compression mechanism for user compression schemes #48