Add an option to copy foreign layers

[mduft]

I would like to skopeo copy a windows image to a linux host to create (via pure filesystem manipulation with umoci) new layers. Windows is dead slow with a lot of files, so I'd rather use linux :) I don't need to be able to run any commands, thus I don't need support for Dockerfile, RUN, etc. I just want to add files, and commit a new layer.

Trying out skopeo to do this, I unfortunately get:

skopeo --override-os windows --override-arch amd64 copy docker://microsoft/windowsservercore:ltsc2016 oci:windows-test:latest
Getting image source signatures
Skipping foreign layer "sha256:3889bb8d808bbae6fa5a33e07093e65c31371bcf9e4c38c21be6b9af52ad1548" copy to oci
Skipping foreign layer "sha256:6631c2d2a60cd7ee5b334c2725b03e4d4976abb9a19c8e8dc9b806b3752745a6" copy to oci
Copying config sha256:9657141ae0624406d5e84631b2cab0eee14d2810a7ca29250d32d1babb68101b
 493 B / 493 B [============================================================] 0s
Writing manifest to image destination
Storing signatures

Any advice? Should this work?

[mtrmac]

Well, in a sense that is probably not all that useful to you, this behavior is correct: the original image had links to foreign layers, OCI can represent links to foreign layers, so the copy also contains correctly copied links to foreign layers.

I do agree that this may not always be desirable, and adding an option to containers/image/copy.Image to make a copy of the foreign layers regardless, and the exposing it in the skopeo CLI, may be warranted.

In the meantime, you can force the existing code to make a copy by using a destination that does not support foreign URLs, e.g. a dir: transport, possibly as an intermediate step:

# $existing_copy_command docker://microsoft/… dir:tmp
# skopeo copy dir:tmp oci:windows-test:latest

[mtrmac]

possibly as an intermediate step:

Oh, intermediating through dir: to oci: does not work either, because the copy to dir: does not rewrite the manifest (that only happens when something about the layers changes during the copy), and the subsequent copy to oci: again finds that it can represent the foreign references and uses them.

So, adding the option to explicitly opt out would be the only clean solution.

[mduft]

Thanks for the answer(s). It at least gives me an idea, despite there is no solution (yet). Thanks :)

[rhatdan]

@mtrmac @vrothberg Is this something we should add? Something an intern could work on?

[vrothberg]

I think this is a good issue for interns, yes.

[rhatdan]

Since this has never been worked on, and not in the interest of the Core Red Hat team that is working on it. It is up to community to make the changes. We would love to have someone from the community add this feature.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[mtrmac]

FWIW the infrastructure for this now exists: copy.Options.DownloadForeignLayers. So all that is necessary is to expose this in the CLI.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rgl]

Dear maintainers, can you please help getting a flag to enable copy.Options.DownloadForeignLayers as mentioned above? Or guide me in getting that flag? :-)

[mtrmac]

Compare how many other copy.Options flags are based in the copy command; I don’t expect this one to be drastically different.

[rgl]

@mtrmac, I gave it a try at https://github.com/rgl/skopeo/commit/481b4baf66faff375d92c4b4989a570845163439 but it does not seem to work:

$ ./bin/skopeo copy --debug --all docker://docker.io/ruilopes/example-docker-buildx-go:v1.6.0  docker://localhost:5000/example-docker-buildx-go:v1.6.0
...
DEBU[0004] PUT http://localhost:5000/v2/example-docker-buildx-go/manifests/sha256:e9907d195e669b89e80c0b940f4d5359c7c4979738d2c764f598dfe13bc9c64a 
DEBU[0004] Error uploading manifest sha256:e9907d195e669b89e80c0b940f4d5359c7c4979738d2c764f598dfe13bc9c64a to localhost:5000/example-docker-buildx-go: received unexpected HTTP status: 500 Internal Server Error while writing manifest "{\n   \"schemaVersion\": 2,\n   \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n   \"config\": {\n      \"mediaType\": \"application/vnd.docker.container.image.v1+json\",\n      \"size\": 1740,\n      \"digest\": \"sha256:7aad7680a864c9f637031aa86fa6d0ad3cc2d9b78d50aca9ced64ff708b1723c\"\n   },\n   \"layers\": [\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.foreign.diff.tar.gzip\",\n         \"size\": 103153235,\n         \"digest\": \"sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26\",\n         \"urls\": [\n            \"https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26\"\n         ]\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 2429631,\n         \"digest\": \"sha256:e9309d65b2001b4a4c7352642f3a7e0e9b1717f695a73e5c731075d9ec7d2d6f\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 1083,\n         \"digest\": \"sha256:1e4e395adf8e39f6a1da5f2d8b6aafd12051e3a9c4304b27b000201e37bb9c30\"\n      }\n   ]\n}" 
DEBU[0004] Writing manifest using preferred type application/vnd.docker.distribution.manifest.v2+json failed: writing manifest: uploading manifest sha256:e9907d195e669b89e80c0b940f4d5359c7c4979738d2c764f598dfe13bc9c64a to localhost:5000/example-docker-buildx-go: received unexpected HTTP status: 500 Internal Server Error 
FATA[0004] copying image 4/5 from manifest list: writing manifest: uploading manifest sha256:e9907d195e669b89e80c0b940f4d5359c7c4979738d2c764f598dfe13bc9c64a to localhost:5000/example-docker-buildx-go: received unexpected HTTP status: 500 Internal Server Error 

The problem seems to be that it does not rewrite the application/vnd.docker.image.rootfs.foreign.diff.tar.gzip layer to application/vnd.docker.image.rootfs.diff.tar.gzip:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1740,
      "digest": "sha256:7aad7680a864c9f637031aa86fa6d0ad3cc2d9b78d50aca9ced64ff708b1723c"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
         "size": 103153235,
         "digest": "sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26",
         "urls": [
            "https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:d555a7e4de4dd775379d5c43c1419374bff7908670dc7444be5e8e8f386f3d26"
         ]
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2429631,
         "digest": "sha256:e9309d65b2001b4a4c7352642f3a7e0e9b1717f695a73e5c731075d9ec7d2d6f"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1083,
         "digest": "sha256:1e4e395adf8e39f6a1da5f2d8b6aafd12051e3a9c4304b27b000201e37bb9c30"
      }
   ]
}

[mtrmac]

@rgl Thanks, for registries, that’s an important point. (The original request was for oci: destinations, where this should not happen.)

Is the registry configured to accept foreign layers? Compare https://github.com/distribution/distribution/blob/2ccf55b8c4c2657a6b8e0873607a69f14d30d557/docs/configuration.md#manifests (this is disabled by default); that might be different for another implementation.

Alternatively, https://github.com/containers/image/pull/1063 speculates about TranslateForeignLayers — that doesn’t exist, and for many users it’s a noticeably worse choice than configuring the registry, because changing manifest breaks digest references (and signatures, if any).

[rgl]

@mtrmac, changing the default registry configuration helped! skopeo copy is no longer failing to upload the manifest! thx for the tip!

This is what I appended to the /etc/docker/registry/config.yml file:

validation:
  manifests:
    urls:
      allow:
        - .+

With that, it seems the registry will now allow any external reference put inside the manifests, but how will the clients deal with that? Will they still try to connect to the external reference? Or will they first try it from the local registry?

[mtrmac]

That’s really up to each individual client’s implementation.

c/image users (incl. Skopeo, Buildah, Podman) prefer the external URLs, in order, and silently fall back to accessing the registry if none of the URLs are accessible.

[rgl]

@mtrmac, got it! thanks for the help! :-)

[mtrmac]

@rgl Would you care to turn that small change into a full PR, with a real CLI option, documentation and the like?

[rgl]

@mtrmac, yes. I will do the PR in the next couple of days. I have to first check whether this actually worked for my use-case.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[michael-sabrnak-swi]

Hi, it seems DownloadForeignLayers feature was never merged and officially published -

I can't find it in any documentation or default branch code (https://github.com/containers/skopeo/blob/main/cmd/skopeo/copy.go#L286), only in @mtrmac fork here: https://github.com/rgl/skopeo/blob/test/cmd/skopeo/copy.go#L265

Is there any plan to merge this?

[mtrmac]

That implementation, hard-coding DownloadForeignLayers to true, is probably not a good default for most Skopeo users; we need to add an option, with end-user documentation.

I think exposing this would be useful, but it’s also not something I’m very likely to work on short-term.

[michael-sabrnak-swi]

I understand. We use an air-gapped environment for Windows images and it's impossible using Foreign Layers so we need to cache them.

I tested your modifications from your test branch on my macOS and it works just for oci: destinations. If I will try to docker:// location (AWS ECR in this case) it says foreign layer exists there (is pushed) but its not really cached there and keeps reference to foreign layer.

Example:

Pulling to local oci: path

./skopeo --version
skopeo version 1.9.2-dev commit: 481b4baf66faff375d92c4b4989a570845163439

./skopeo copy --insecure-policy --debug \
    docker://docker.io/jetbrains/teamcity-agent:2022.10.3-windowsservercore-1809 \
    oci:teamcity-agent_2022.10.3-windowsservercore-1809

...
DEBU[0002] GET https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f
...
DEBU[0002] GET https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:b5f4ee3f051b6848a99134adfb17dc9f8f81284bff92912318397b995ffc71c7
...
Copying blob 8185ee4ed646 [--------------------------------------] 8.0b / 1.5GiB
...
Copying blob b5f4ee3f051b [--------------------------------------] 8.0b / 285.2MiB
...

Size is correct with 5.3G in local FS

❯ du -h teamcity-agent_2022.10.3-windowsservercore-1809
5.3G    teamcity-agent_2022.10.3-windowsservercore-1809/blobs/sha256
5.3G    teamcity-agent_2022.10.3-windowsservercore-1809/blobs
5.3G    teamcity-agent_2022.10.3-windowsservercore-1809

Pushing from local oci: path to remote docker:// ECR registry

./skopeo copy --insecure-policy --debug \
    --dest-creds='AWS:PWD' \
    oci:teamcity-agent_2022.10.3-windowsservercore-1809 \
    docker://XXX.dkr.ecr.us-east-1.amazonaws.com/XXX/XXX:teamcity-agent_2022.10.3-windowsservercore-1809

...
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Checking if we can reuse blob sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.nondistributable.v1.tar+gzip" = true
DEBU[0000] Checking /v2/XXX/blobs/sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f
DEBU[0000] GET https://XXX.dkr.ecr.us-east-1.amazonaws.com/v2/
DEBU[0000] Ping https://XXX.dkr.ecr.us-east-1.amazonaws.com/v2/ status 401
DEBU[0000] HEAD https://XXX.dkr.ecr.us-east-1.amazonaws.com/v2/XXX/blobs/sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f
DEBU[0001] ... already exists
DEBU[0001] Skipping blob sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f (already present):
Copying blob 8185ee4ed646 skipped: already exists
DEBU[0001] Checking if we can reuse blob sha256:b5f4ee3f051b6848a99134adfb17dc9f8f81284bff92912318397b995ffc71c7: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.nondistributable.v1.tar+gzip" = true
DEBU[0001] Checking /v2/XXX/blobs/sha256:b5f4ee3f051b6848a99134adfb17dc9f8f81284bff92912318397b995ffc71c7
Copying blob 8185ee4ed646 skipped: already exists
DEBU[0002] ... already exists
Copying blob 8185ee4ed646 skipped: already exists
Copying blob b5f4ee3f051b skipped: already exists

However size is 3.7G in remote registry aws ecr list-images --repository-name XXX

...
        {
            "registryId": "XXX",
            "repositoryName": "XXX/XXX",
            "imageDigest": "sha256:da460c86bfd800ed39cf50216d80f661def131ed4f0abca387254737f8b8e96a",
            "imageTags": [
                "teamcity-agent_2022.10.3-windowsservercore-1809"
            ],
            "imageSizeInBytes": 3711704672,
            "imageManifestMediaType": "application/vnd.oci.image.manifest.v1+json",
            "artifactMediaType": "application/vnd.oci.image.config.v1+json"
        }
...

I checked manifest file and it keeps reference of foreign layer URL:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:bcdf8863158a16afb1ea8683a2d15232ab6222c2dc780b16da6ffff218cf7d4b",
    "size": 18860
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.nondistributable.v1.tar+gzip",
      "digest": "sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f",
      "size": 1660377526,
      "urls": [
        "https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f"
      ]
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nondistributable.v1.tar+gzip",
      "digest": "sha256:b5f4ee3f051b6848a99134adfb17dc9f8f81284bff92912318397b995ffc71c7",
      "size": 299030232,
      "urls": [
        "https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:b5f4ee3f051b6848a99134adfb17dc9f8f81284bff92912318397b995ffc71c7"
      ]
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:c4572ee3b02fe9d2c369bdfb6048fd56c77993e36a90b99505c977fc068eb1fa",
      "size": 1294
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4cc4488f95ebd61aaf54d914d7494ac28fc2b29b727ee34cbc7c1ec9bf4d517f",
      "size": 929851592
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:1a1a339cf68ec147687f93802d3caec1fe6431679b322c2caeddcc15b826078c",
      "size": 1294
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b1c4cab7f7704d51e03f0658f826bac24098c9538e95252777dbfc90cf0f0623",
      "size": 7967696
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:8d0a805f564f27b23b8ad9a3a3918e8dec428ec5f92dfab596a689bf88a38719",
      "size": 1764409328
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4d93feabc2e7c7d7ef6adb43f4c4e7893fc18c75b74804615faebdc69ae0aae0",
      "size": 611945
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:6ed54d6cab6f675caa24d18dd31d76c6d04c5da95db467fb3bbb743b45a8e091",
      "size": 180958271
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:04dfddc0c59cfb4e74f862c9a844049d978bf2821a8173a766155311a9e9e7b3",
      "size": 1294
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d958746d90555b7ba51781528627379167d6953a89881b01baec4a4dceb1a2eb",
      "size": 4213
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:011a394f228548b7d1516a14f94eca786d5b4cf202b21a36dd4067f9f94c9d52",
      "size": 1414
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:88f65d9974d48120010728b2888a1c87d75529d69f14d8173724c00d96f82afa",
      "size": 1363
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d471bca265282c700b0061a2f9a2732f9d179a5f06b844f6f35ffb235781676d",
      "size": 1393
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f144e73cd6c71007a725e2c69d48b320348a63a7e458d58c6254563d54ef4f7a",
      "size": 1397
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:a0b8002dee3b4784edbca66df078f15346e11755351b6347fa784086c8192971",
      "size": 1370
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:5094feb8a5200de109e2c2cc8c7dd570d329e39495cf3aa1469fbb44e1d37d72",
      "size": 1402
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:e7ffe87632ff77a11c3f7b832040051422ffbd4927b9647bba4e128d605b6a7d",
      "size": 1404
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:bffc7e1af625bb3c3c9fc22d7652b9c8639687324df5037a37de79723af3e3c7",
      "size": 1390
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:ddbe3b3ac5e065b8708126c7c7e609c9428dfcd7d8159aa3322d1cf0ce0d1103",
      "size": 1355
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:040d41d23f9733b33adee4daf46fbadec4fec3d709b404c1fe4cd6c331101ac7",
      "size": 1423
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:711541876a0d48a21e7ce6f7e8b060fe5aeaf593b6d094c931a52b82f95a307a",
      "size": 1408
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:467057ff8dd116b394b17430facd458d05874508fca700936b2b67db4abd4393",
      "size": 1416
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:7e5b2e371f5f914f477d0bfb3639d412312b6bb6bf4f5d1fa2bd199f4a068386",
      "size": 535406815
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b7614c95d5f80b15ea2629ea043e74a8e414c857d4fe0d1fac17dc45a5a91854",
      "size": 292223839
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:858487029e4452ecd2681865a03d62ed6118d4b24e8c0e5359b93945aeca1549",
      "size": 1383
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:9e7986c0b22e1e66bb535ffec4b8e767e7a167f2f7ad42e72d543890038f8072",
      "size": 1373
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:44303858e3a15fe95d774b5bee8dd4591441caed135bf63c182c627a2184344d",
      "size": 1398
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:72ccf5840e6caa63028b8a3ef4aa038670d3a6a084a38409bcc7656d6ebc08b8",
      "size": 1360
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:e03d307f7e12934e776e142cc9fdd24b8eab324dd8b6089b9e6bf4098fe25cbb",
      "size": 1396
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:903c9657c0e09d15be645b8a85c99e971269a7440bd1acc817b1ebb6b5259c2b",
      "size": 223210
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d3dc526a56506643a2a0157ac5f7afe51dbbfefea967ce565bdbdaf9995517f8",
      "size": 1376
    }
  ]
}

When I tried to modify and remove urls reference it still did not uploaded foreign layers to ECR registry.

So in conclusion this works just for oci paths but I guess it's up to the remote registry how this will behave.

Do you know how I can force the upload of the foreign layers to the remote registry? Thanks.

Update

I managed to push image to ECR registry. But it required adjustment of manifest file:

From:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:bcdf8863158a16afb1ea8683a2d15232ab6222c2dc780b16da6ffff218cf7d4b",
    "size": 18860
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.nondistributable.v1.tar+gzip",
      "digest": "sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f",
      "size": 1660377526,
      "urls": [
        "https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f"
      ]
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nondistributable.v1.tar+gzip",
      "digest": "sha256:b5f4ee3f051b6848a99134adfb17dc9f8f81284bff92912318397b995ffc71c7",
      "size": 299030232,
      "urls": [
        "https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:b5f4ee3f051b6848a99134adfb17dc9f8f81284bff92912318397b995ffc71c7"
      ]
    },
...

To:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:bcdf8863158a16afb1ea8683a2d15232ab6222c2dc780b16da6ffff218cf7d4b",
    "size": 18860
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:8185ee4ed6467d79d4c69f02fe3902311ba84777802d459dcd99869f603b161f",
      "size": 1660377526
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b5f4ee3f051b6848a99134adfb17dc9f8f81284bff92912318397b995ffc71c7",
      "size": 299030232
    },
...

Result:

aws ecr list-images --repository-name XXX

...
        {
            "registryId": "XXX",
            "repositoryName": "XXX/XXX",
            "imageDigest": "sha256:691e6b9bfb8cae45fb0ab88840bb4af3b8da302c30fb7907c6557c69bdef2b84",
            "imageTags": [
                "teamcity-agent_2022.10.3-windowsservercore-1809"
            ],
            "imageSizeInBytes": 5671112430,
            "imageManifestMediaType": "application/vnd.oci.image.manifest.v1+json",
            "artifactMediaType": "application/vnd.oci.image.config.v1+json"
        }
...

So as expected digest will be different because manifest is adjusted but we have the fully offline image stored in ECR prepared for the air-gapped environment.

So workflow was as following:

• Pull image as oci:
• Adjust manifest file (replace nondistributable and remove URL reference)
• Push image from oci: path to docker:// remote registry path.

[mtrmac]

If the registry says that the blob is already there, then that’s not something the client can change — even if the client uploaded it, the registry would again say that the blob is already there :) Assuming the blob can actually be downloaded from that registry, that’s anyway exactly the outcome you wanted.

It seems to me that the only clear issue in the above, if any, is that the aws ecr list-images output does not contain the the total size you expect, maybe by explicitly filtering out non-distributable layers. It’s not at all clear to me whether that’s a reliable indication whether that data is actually present on the registry or not; my naive assumption is that the data is actually present, and this is just the way the size is computed. But that would need to be verified by actually pulling the image in some way that prevents use of the external URLs.

[michael-sabrnak-swi]

@mtrmac I tested the behavior and yes the blobs were there indeed but the image was constructed based on a manifest file from both local and foreign layers and not from local layers (that's why size was different).

So the manual change I mentioned:

From: "mediaType": "application/vnd.oci.image.layer.nondistributable.v1.tar+gzip" To: "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip"

And possibly removal of urls helped to see the real size (from the AWS ECR repository and from the client side where I pulled image later didn't contact mcr.microsoft.com servers) - verified via hosts file with blocked access to real Microsoft servers servers.