Add support for resuming downloads that were cancelled/hit an error in an earlier invocation. Doing this required implementing our own Pull routine, separate from oras.Copy, which is used elsewhere.
If a server responds with header Accept-Ranges: "bytes", treat the download as resumable; otherwise, pull behaves as before. For resumable downloads:
Files are saved to $KITOPS_HOME/storage/ingest/<blob-digest>
If a file exists, we first read it to get a digest from the start to where the download ended, then seek the download reader to that point and resume downloading
Once completed, we verify the digest and move it to the normal location
There's some risk of corruption if e.g. two separate kit pulls are pulling the same resumable download simultaneously, though I think the risk is fairly low and I'm not sure we care to support running kit in parallel at this moment.
When a pull is completed, all data in the ingest directory is removed. This is to ensure we don't leave files lying around with no easy path to their removal, which can happen with non-resumable downloads that are e.g. cancelled via ctrl-C. This means that completing a pull effectively cancels all other downloads.
Finally, since I had to add reimplement concurrency for the pull operation, I've added a --concurrency flag to allow configuring it. It's available on all commands that take the standard network options, but is currently only used for pull and push.
Description
Add support for resuming downloads that were cancelled/hit an error in an earlier invocation. Doing this required implementing our own Pull routine, separate from oras.Copy, which is used elsewhere.
If a server responds with header
Accept-Ranges: "bytes", treat the download as resumable; otherwise, pull behaves as before. For resumable downloads:$KITOPS_HOME/storage/ingest/<blob-digest>There's some risk of corruption if e.g. two separate kit pulls are pulling the same resumable download simultaneously, though I think the risk is fairly low and I'm not sure we care to support running
kitin parallel at this moment.When a pull is completed, all data in the ingest directory is removed. This is to ensure we don't leave files lying around with no easy path to their removal, which can happen with non-resumable downloads that are e.g. cancelled via ctrl-C. This means that completing a pull effectively cancels all other downloads.
Finally, since I had to add reimplement concurrency for the pull operation, I've added a
--concurrencyflag to allow configuring it. It's available on all commands that take the standard network options, but is currently only used for pull and push.Linked issues
Closes #311 Closes #387