search

nlewo / nix2container

An archive-less dockerTools.buildImage implementation
Apache License 2.0
501 stars 45 forks source link

skopeo copy always copies a layer, even if the hash is the same #121

Closed iteratee closed 5 months ago

iteratee commented 6 months ago

I am seeing skopeo copy with a nix: source always copy a layer, even if the hash is the same.

Looking at the code, I suspect that it may be related to not returning the blob size. I understand that returning the size was causing an issue with skopeo. I was getting a different error about closed pipes, and was able to clear it by using a newer nixpkgs from nixpkgs-unstable, with a newer skopeo to be patched.

It may be worth checking if a newer skopeo still has a problem when the blob size is returned.

nlewo commented 6 months ago

@iteratee IIRC, i only observed this behavior when copying to the Docker deamon. When copying layers to Podman or a registry, Skopeo was able to skip already pushed layers.

@iteratee What is your destination Skopeo transport? Is it docker-daemon?

iteratee commented 6 months ago

@iteratee IIRC, i only observed this behavior when copying to the Docker deamon. When copying layers to Podman or a registry, Skopeo was able to skip already pushed layers.

@iteratee What is your destination Skopeo transport? Is it docker-daemon?

The transport is docker://. Is Skopeo relying on some local cache? Because this occurs in CI, which wouldn't have that local cache populated. They are being pushed to ECR. I found two recent CI runs where all but the last 4 layers have the same hash, and there is no indication that they are being skipped.

nlewo commented 6 months ago

Is Skopeo relying on some local cache?

I don't think so. IIRC, Skopeo gets the layer hash and query the registry to know if this hash is already present.

there is no indication that they are being skipped.

Hm, i think Skopeo should mention it on its output. Maybe something changed in Skopeo...

Maybe you could try to reproduce locally by spawning a Docker registry. It was pretty trivial to spawn a Docker registry with docker run.

iteratee commented 6 months ago

I will try to start up a docker registry on Monday.

iteratee commented 5 months ago

Thank you for your help. It turns out that the output that skopeo produces when run without a tty doesn't indicate if a layer was copied or if the copy was skipped. When testing locally with a docker image registry, it does skip layers it already has. Even when skipping, the non-tty output looks like this:

Copying blob sha256:bf383fd228100e9f091b8803e5062cae2e9296c179005af423365bc485df8830
Copying blob sha256:4c5e46b043941ebe259109b3f55d3339b561bb4daf7f83248951cd46ae5debcd
nlewo commented 5 months ago

For the reference, there is the following related Skopeo opened issue: https://github.com/containers/skopeo/issues/658