search

cirruslabs / tart

macOS and Linux VMs on Apple Silicon to use in CI and other automations
https://tart.run
Other
3.65k stars 102 forks source link

Copy images between registries / Error: invalidData #833

Closed remarkov closed 3 days ago

remarkov commented 1 month ago

Probably more a question, not an issue. I hope this can be useful for others too.

When having more than one local image registry in the network, what would be the best way to copy/sync images between the registries without tart pull and further tart push? So far I tried using Skopeo which works most of the times but sometimes the copied image gets corrupt and tart clone or tart pull give an error during the progress

pulling disk (160.6 GB compressed)...
9%
Error: invalidData

Most likely this is because Tart image format is not fully compatible with Docker images but that's just a guess. Any advice will be much appreciated.

fkorotkov commented 1 month ago

Do you have a hard requirement of having different registries? Have you considered a single registry with several read through proxies?

edigaryev commented 1 month ago

Error: invalidData

Which Tart version are you using to generate the VM images?

This error seems to be coming from the LZ4 decompressor and could be caused by the Skopeo reordering the manifest layers, which is an issue in the older V1 disk format.

remarkov commented 1 month ago

thanks for quick response!

@fkorotkov yes, unfortunately we need separate registries located in different locations/networks but some may share some of the images, not necessarily all.

@edigaryev that's on Tart 0.38.0. Do you by any chance know some workaround to prevent the layers reordering? Does it make sense to try other tools like regclient or this is something Tart-specific?

fkorotkov commented 1 month ago

We don't have experience with Skopeo or similar tools. Some other tools in our experience were Docker exclusive and not fully OCI compliment until the latest versions.

Another concerning part is that this issue is transient and only sometimes things getting corrupted. Definitely points to something fishy in the tooling. Hope you'll figure it out!

fkorotkov commented 1 month ago

I just doble checked and Disk V2 also depends on ordering but we can fix it in backward compatible way.

edigaryev commented 1 month ago

@remarkov I've tried reproducing this using a simple script that does tart push, skopeo copy and tart pull in a loop to no avail on the latest Tart version (2.11.0). So the initial premise that this issue is related to layer order is probably wrong.

Could try pulling your corrupted image using the latest Tart to see if the issue still persists? Ideally, push a new VM image using the tart push to get the newer V2 disk image.

It has countless improvements over the older version, including the V2 disk images. We've also just merged https://github.com/cirruslabs/tart/pull/835 to nail this error further down if it ever occurs again.

remarkov commented 1 month ago

@edigaryev Thanks for your time, I will try to do the tests with the newer version too.