search

AustralianAntarcticDataCentre / raadsync

Other
2 stars 0 forks source link

libarchive #40

Closed mdsumner closed 7 years ago

mdsumner commented 7 years ago

https://github.com/jimhester/archive

raymondben commented 7 years ago

Yeehar, Z-compressed files here we come!

raymondben commented 7 years ago

Hmm. Maybe. The underlying (compressed) file still needs to be a supported archive format (https://github.com/libarchive/libarchive/wiki/LibarchiveFormats). So Z-compressed binary or netCDF don't seem to work out of the box. Maybe we can leverage just the Z-decompression (e.g. https://github.com/libarchive/libarchive/wiki/FormatRaw) but I'm not sure this is possible through Jim's bindings yet.

mdsumner commented 7 years ago

I was thinking of the decompression to stand-alone, not streaming from the archive - this is still a step up from system calls right?

raymondben commented 7 years ago

Sure, I just don't see how to do it (yet)!

raymondben commented 7 years ago

Update: now have file_read which can be used to decompress .Z files:

ff <- file_read("20160101.nc.Z")
writeBin(readBin(ff,"raw",10e7),"20160101.nc")
close(ff)
raymondben commented 7 years ago

Implemented: repo config entries can now specify "uncompress" or "uncompress_delete" as a postprocessing step. The CERSAT sea ice default definition has been updated to use this.