search

open-component-model / ocm-project

OCM Project Backlog
Apache License 2.0
0 stars 0 forks source link

Enhance `ocm transfer` to be able to be restarted from where it left of in case of an interruption #36

Open morri-son opened 1 year ago

morri-son commented 1 year ago

What would you like to be added: ocm transfer, especially when dealing with product components and large images, currently needs to be started from scratch in case the transfer is interrupted. The transfer should be improved in a way that only artifacts that a broken transfer can be restarted and then only artifacts that have not already have been transferred are transported. Why is this needed: Large transports can take a large amount of time. In case of issues during a transfer, it would be good if a subsequent transfer can be shortened and concentrate on the missing artifacts.

Skarlso commented 4 months ago

This is an interesting one actually. It might be something that we could do in case of HTTP based operations using the TUS protocol https://tus.io/.

But in case of OCI registries and clients hiding HTTP header manipulations ( looking at you github client ) it would be difficult to perform a seek operation. Even if we do, most of these network operations don't allow seeking. So we would most likely, have to read all bytes again anyways. The only thing we would save for OCI based registry transfer would be that some of the bytes we don't have to write again.

Which might be okay if the destination hardware is slow. So it's worth looking into it.