Make the gzipped tarball output rsyncable for poor man's delta updates

afbjorklund commented 5 months ago

What is the problem you're trying to solve

Makefile: gzip -9 $(CURDIR)/_output/nerdctl-full-$(VERSION_TRIMMED)-linux-$(1).tar

Describe the solution you'd like

Add the --rsyncable flag to the gzip invocation above, to make the output more rsync friendly.

Additional context

No response

AkihiroSuda commented 5 months ago

👍

(Off-topic: --rsyncable might be also useful for dedupe of OCI image layer blobs?)

afbjorklund commented 5 months ago

If you want it accessible over HTTP as well, then we should also create a .zsync file.

http://zsync.moria.org.uk/

$ zsync -i nerdctl-full-1.7.3-linux-arm64.tgz http://localhost/zsync/nerdctl-full-1.7.5-linux-arm64.tgz.zsync
#################### 100.0% 0.0 kBps DONE    

reading seed file nerdctl-full-1.7.3-linux-arm64.tgz: ********************************************************************************************************************
Read nerdctl-full-1.7.3-linux-arm64.tgz. Target 33.3% complete.      ************************************
downloading from http://localhost/zsync/nerdctl-full-1.7.5-linux-arm64.tgz:
#################### 100.0% 0.0 kBps DONE    

verifying download...checksum matches OK
used 74076160 local, fetched 148500742
$ zsync -i nerdctl-full-1.7.4-linux-arm64.tgz http://localhost/zsync/nerdctl-full-1.7.5-linux-arm64.tgz.zsync
#################### 100.0% 0.0 kBps DONE    

reading seed file nerdctl-full-1.7.4-linux-arm64.tgz: ********************************************************************************************************************
Read nerdctl-full-1.7.4-linux-arm64.tgz. Target 37.1% complete.      ************************************
downloading from http://localhost/zsync/nerdctl-full-1.7.5-linux-arm64.tgz:
#################### 100.0% 0.0 kBps DONE    

verifying download...checksum matches OK
used 82632704 local, fetched 139945483

Where .tgz here was the .tar.gz, but re-compressed with gzip --rsync --best

Does require (lib)zsync, but...

afbjorklund commented 5 months ago

As advertised, the overhead is around 1% 210M nerdctl-full-1.7.5-linux-arm64.tar.gz 213M nerdctl-full-1.7.5-linux-arm64.tgz 372K nerdctl-full-1.7.5-linux-arm64.tgz.zsync

I don't think this is worthwhile for the small tarballs. Also, they include a new binary version every time.

HTTPS support: https://github.com/probonopd/zsync-curl

containerd / nerdctl