lets pick an archive format

[max-mapper]

we wanna support an archive format for storing + exchanging compressed files. in our specific use case the files will be container filesystems (e.g. a debootstrapped ubuntu.iso), as well as a set of research data (csvs etc) that might be used in an analysis pipeline

two features we want:

• format supports random access on compressed archives
• try and use something that exists and is on peoples computers already (e.g. they can double click the thing to extract it)

some candidates:

• zip
• something like tar or xz with block compression and an external file index for files -> byte offsets

research:

• http://asdf-standard.readthedocs.org/en/latest/
• https://wiki.freedesktop.org/www/Software/systemd/Particle/
• https://github.com/stevvooe/continuity
• https://github.com/opencontainers/specs/issues/11
• https://github.com/vasi/pixz
• https://github.com/ipfs/archive-format/issues/7
• http://libguestfs.org/virt-sparsify.1.html

[wohali]

Another thing worth looking at: https://github.com/fidlej/idzip

[onbjerg]

pifs? :new_moon_with_face:

[max-mapper]

@mrocklin reminded me to look into dar http://dar.linux.free.fr/doc/man/dar.html, seems pretty good so far

[max-mapper]

some initial cons of dar:

• libdar is 2.7mb of c++ code
• development is still on sourceforge
• seems quite complex, other independent implementations would likely be very difficult (contrast this to tar which is quite easy to implement)

[ekg]

Another con is that it doesn't seem widely used. On Nov 29, 2015 10:59 PM, "maxogden" notifications@github.com wrote:

some initial cons of dar:

• libdar is 2.7mb of c++ code
• development is still on sourceforge
• seems quite complex, other independent implementations would likely be very difficult (contrast this to tar which is quite easy to implement)

— Reply to this email directly or view it on GitHub https://github.com/datproject/discussions/issues/27#issuecomment-160475037 .