docker/internal/tarfile has incorrect media type in manifests; copy layer format is not manifest-format dependent

When Skopeo copies an image to our Artifactory Docker repo the first layer 5f70bf18a086 is not compressed but has type gzip in the manifests. I noticed during the copy it sees that the first layer is already in Artifactory so doesn't copy it again but it's the uncompressed SHA, not sure if that is related? But I'd expect it to compress the layer and upload that layer instead of using the already existing uncompressed one.

Skopeo Version: skopeo version 1.3.0

Skopeo copy command:

skopeo copy --debug --retry-times 3 --dest-creds ****:**** --sign-by <redacted> docker-daemon:<local_image> docker://<image>

Logs:

time="2021-06-22T13:11:45Z" level=debug msg="... already exists"
time="2021-06-22T13:11:45Z" level=debug msg="Skipping blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef (already present):"
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef

$ skopeo inspect --raw --creds "${SKOPUSER}":"${SKOPPASS}" docker://<image> | jq

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 4686,
    "digest": "sha256:47d8cc97b2c309403ac13968d50706e14daa7e89868f3cb391640d1ac38cc729"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1024,
      "digest": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 15192304,
      "digest": "sha256:1850fe5b1c679d9e5b5557cafe9843c3ce251183e901fc4c169608aab29a3259"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 556,
      "digest": "sha256:5e8dcaf7687f5c9cf34b3c213590668c985242da53291b4131fd85e8a4d4c421"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 180,
      "digest": "sha256:15d2f5b3eaa2cbcde1db1bc8e99e2d777e6d19714e64e0e768f2dc19f157eede"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 176,
      "digest": "sha256:06f696127abebdbd4d7f50c869214a14ac1ab0a5ad836e5b11bd7f6b3ed23293"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 403,
      "digest": "sha256:044c1498d2c73ed0dfb934a18ca2d3575f4a7ac09dd171173b990800e7faa3cc"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 315,
      "digest": "sha256:ce12853b59f49dd81d63b2a3c3362af6a7f284f0b16accd2194d0ae28dd76b82"
    }
  ]
}

Downloaded Manifest file:

skopeo   copy --src-tls-verify=true --src-creds "${SKOPUSER}":"${SKOPPASS}" --dest-oci-accept-uncompressed-layers docker://<image> oci:/tmp/foobar/raw:image
cd /tmp/foobar/raw/blobs/sha256/
cat 666520ec7cea326046be6afa76b1b25b0a9f618ff18b13c9e89a59f1fb395806 | jq

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:d1f06d5ae426406b9a4828dca9987c92b65b3fbb76b979cea0bb35bdad3b085d",
    "size": 2922
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
      "size": 1024
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:1850fe5b1c679d9e5b5557cafe9843c3ce251183e901fc4c169608aab29a3259",
      "size": 15192304
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:5e8dcaf7687f5c9cf34b3c213590668c985242da53291b4131fd85e8a4d4c421",
      "size": 556
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:15d2f5b3eaa2cbcde1db1bc8e99e2d777e6d19714e64e0e768f2dc19f157eede",
      "size": 180
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:06f696127abebdbd4d7f50c869214a14ac1ab0a5ad836e5b11bd7f6b3ed23293",
      "size": 176
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:044c1498d2c73ed0dfb934a18ca2d3575f4a7ac09dd171173b990800e7faa3cc",
      "size": 403
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:ce12853b59f49dd81d63b2a3c3362af6a7f284f0b16accd2194d0ae28dd76b82",
      "size": 315
    }
  ]
}

5f70bf18a086 is not compressed

$ file *
044c1498d2c73ed0dfb934a18ca2d3575f4a7ac09dd171173b990800e7faa3cc: gzip compressed data, original size modulo 2^32 3072
06f696127abebdbd4d7f50c869214a14ac1ab0a5ad836e5b11bd7f6b3ed23293: gzip compressed data, original size modulo 2^32 3072
15d2f5b3eaa2cbcde1db1bc8e99e2d777e6d19714e64e0e768f2dc19f157eede: gzip compressed data, original size modulo 2^32 3072
1850fe5b1c679d9e5b5557cafe9843c3ce251183e901fc4c169608aab29a3259: gzip compressed data, original size modulo 2^32 39132672
5e8dcaf7687f5c9cf34b3c213590668c985242da53291b4131fd85e8a4d4c421: gzip compressed data, original size modulo 2^32 5632
5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef: data
666520ec7cea326046be6afa76b1b25b0a9f618ff18b13c9e89a59f1fb395806: JSON data
ce12853b59f49dd81d63b2a3c3362af6a7f284f0b16accd2194d0ae28dd76b82: gzip compressed data, original size modulo 2^32 2560
d1f06d5ae426406b9a4828dca9987c92b65b3fbb76b979cea0bb35bdad3b085d: JSON data

Thanks for your report.

I noticed during the copy it sees that the first layer is already in Artifactory so doesn't copy it again but it's the uncompressed SHA, not sure if that is related? But I'd expect it to compress the layer and upload that layer instead of using the already existing uncompressed one.

No, using the existing one is what the code currently intends to do.

Note to self: It looks like docker/internal/tarfile just always uses the gzip media type (and, to be fair, the uncompressed one is not officially defined at https://github.com/distribution/distribution/blob/main/docs/spec/manifest-v2-2.md ). It’s not trivially clear that this is safe to change.

Either way, please collect the full debug log to show what exactly happens here.

@mtrmac Full Debug Logs:

+ skopeo copy --debug --retry-times 3 --dest-creds ****:**** --sign-by 9E41D6AEBFC84271A81976A9FA1AD8593A6AEAA9 docker-daemon:microservice-ubi-micro:20210622-165754 docker://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/microservice-ubi-micro:20210622-165754
time="2021-06-22T17:00:09Z" level=debug msg="Returning credentials for sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com from DockerAuthConfig"
time="2021-06-22T17:00:09Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
time="2021-06-22T17:00:09Z" level=debug msg=" Using \"docker\" namespace sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com"
time="2021-06-22T17:00:09Z" level=debug msg="  Using file:///tmp/sigstore"
time="2021-06-22T17:00:09Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com"
time="2021-06-22T17:00:09Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2021-06-22T17:00:09Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
time="2021-06-22T17:00:10Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:11Z" level=debug msg="Using blob info cache at /home/jenkins/.local/share/containers/cache/blob-info-cache-v1.boltdb"
time="2021-06-22T17:00:11Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:11Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:11Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:11Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:11Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:11Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:11Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:11Z" level=debug msg="IsRunningImageAllowed for image docker-daemon:docker.io/library/microservice-ubi-micro:20210622-165754"
time="2021-06-22T17:00:11Z" level=debug msg=" Using transport \"docker-daemon\" policy section \"\""
time="2021-06-22T17:00:11Z" level=debug msg=" Requirement 0: allowed"
time="2021-06-22T17:00:11Z" level=debug msg="Overall: allowed"
Getting image source signatures
time="2021-06-22T17:00:11Z" level=debug msg="Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json]"
time="2021-06-22T17:00:11Z" level=debug msg="... will first try using the original manifest unmodified"
time="2021-06-22T17:00:11Z" level=debug msg="Checking /v2/microservice-ubi-micro/blobs/sha256:b65b4568b3d9fccab93853a8917303cdc32cd84d340692905741b0f0040f9be6"
time="2021-06-22T17:00:11Z" level=debug msg="GET https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/"
time="2021-06-22T17:00:11Z" level=debug msg="Checking /v2/microservice-ubi-micro/blobs/sha256:e80007124be152d935f4987552ca5c442e5e9587fd9be09bed52d49387937a3c"
time="2021-06-22T17:00:11Z" level=debug msg="Checking /v2/microservice-ubi-micro/blobs/sha256:bbdd84d406563df3e3e61883d4ffaf781e61b2c31f6011cf086bf77c6ecf4611"
time="2021-06-22T17:00:11Z" level=debug msg="Checking /v2/microservice-ubi-micro/blobs/sha256:45e7ebcfcedd1c136c2b7f5f727c178e7e0987c5c60d12bc83ed1c3155bc9c9b"
time="2021-06-22T17:00:11Z" level=debug msg="Checking /v2/microservice-ubi-micro/blobs/sha256:acd6d8adf51a3bebfa5ab9d1efeb2e7e909e3f289c0409fd8d9dcce0aa1a5656"
time="2021-06-22T17:00:11Z" level=debug msg="Checking /v2/microservice-ubi-micro/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
time="2021-06-22T17:00:11Z" level=debug msg="Ping https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/ status 401"
time="2021-06-22T17:00:11Z" level=debug msg="GET https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/artifactory/api/docker/sc-shared-services-team-dev-docker-local/v2/token?account=cicd%40us.ibm.com&scope=repository%3Amicroservice-ubi-micro%3Apull%2Cpush&service=sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com"
time="2021-06-22T17:00:11Z" level=debug msg="GET https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/artifactory/api/docker/sc-shared-services-team-dev-docker-local/v2/token?account=cicd%40us.ibm.com&scope=repository%3Amicroservice-ubi-micro%3Apull%2Cpush&service=sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com"
time="2021-06-22T17:00:11Z" level=debug msg="GET https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/artifactory/api/docker/sc-shared-services-team-dev-docker-local/v2/token?account=cicd%40us.ibm.com&scope=repository%3Amicroservice-ubi-micro%3Apull%2Cpush&service=sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com"
time="2021-06-22T17:00:11Z" level=debug msg="GET https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/artifactory/api/docker/sc-shared-services-team-dev-docker-local/v2/token?account=cicd%40us.ibm.com&scope=repository%3Amicroservice-ubi-micro%3Apull%2Cpush&service=sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com"
time="2021-06-22T17:00:11Z" level=debug msg="GET https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/artifactory/api/docker/sc-shared-services-team-dev-docker-local/v2/token?account=cicd%40us.ibm.com&scope=repository%3Amicroservice-ubi-micro%3Apull%2Cpush&service=sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com"
time="2021-06-22T17:00:11Z" level=debug msg="GET https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/artifactory/api/docker/sc-shared-services-team-dev-docker-local/v2/token?account=cicd%40us.ibm.com&scope=repository%3Amicroservice-ubi-micro%3Apull%2Cpush&service=sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com"
time="2021-06-22T17:00:12Z" level=debug msg="HEAD https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/sha256:bbdd84d406563df3e3e61883d4ffaf781e61b2c31f6011cf086bf77c6ecf4611"
time="2021-06-22T17:00:12Z" level=debug msg="HEAD https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/sha256:45e7ebcfcedd1c136c2b7f5f727c178e7e0987c5c60d12bc83ed1c3155bc9c9b"
time="2021-06-22T17:00:12Z" level=debug msg="HEAD https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/sha256:b65b4568b3d9fccab93853a8917303cdc32cd84d340692905741b0f0040f9be6"
time="2021-06-22T17:00:12Z" level=debug msg="HEAD https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
time="2021-06-22T17:00:12Z" level=debug msg="HEAD https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/sha256:e80007124be152d935f4987552ca5c442e5e9587fd9be09bed52d49387937a3c"
time="2021-06-22T17:00:12Z" level=debug msg="HEAD https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/sha256:acd6d8adf51a3bebfa5ab9d1efeb2e7e909e3f289c0409fd8d9dcce0aa1a5656"
time="2021-06-22T17:00:12Z" level=debug msg="... already exists"
time="2021-06-22T17:00:12Z" level=debug msg="Skipping blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef (already present):"
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
time="2021-06-22T17:00:12Z" level=debug msg="Checking /v2/microservice-ubi-micro/blobs/sha256:69d13cbd148f8ce0f721ecfc21192cd7e5ca75fab2184fac7a90a3aa151cd8e4"
time="2021-06-22T17:00:12Z" level=debug msg="HEAD https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/sha256:69d13cbd148f8ce0f721ecfc21192cd7e5ca75fab2184fac7a90a3aa151cd8e4"
time="2021-06-22T17:00:12Z" level=debug msg="... not present"
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
Copying blob sha256:bbdd84d406563df3e3e61883d4ffaf781e61b2c31f6011cf086bf77c6ecf4611
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:12Z" level=debug msg="Compressing blob on the fly"
time="2021-06-22T17:00:12Z" level=debug msg="Uploading /v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:12Z" level=debug msg="POST https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:12Z" level=debug msg="... not present"
time="2021-06-22T17:00:12Z" level=debug msg="... not present"
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
Copying blob sha256:e80007124be152d935f4987552ca5c442e5e9587fd9be09bed52d49387937a3c
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:12Z" level=debug msg="Compressing blob on the fly"
time="2021-06-22T17:00:12Z" level=debug msg="Uploading /v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:12Z" level=debug msg="POST https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/"
Copying blob sha256:45e7ebcfcedd1c136c2b7f5f727c178e7e0987c5c60d12bc83ed1c3155bc9c9b
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:12Z" level=debug msg="Compressing blob on the fly"
time="2021-06-22T17:00:12Z" level=debug msg="Uploading /v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:12Z" level=debug msg="POST https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:12Z" level=debug msg="... not present"
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
Copying blob sha256:b65b4568b3d9fccab93853a8917303cdc32cd84d340692905741b0f0040f9be6
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:12Z" level=debug msg="Compressing blob on the fly"
time="2021-06-22T17:00:12Z" level=debug msg="Uploading /v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:12Z" level=debug msg="POST https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:12Z" level=debug msg="... not present"
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
Copying blob sha256:acd6d8adf51a3bebfa5ab9d1efeb2e7e909e3f289c0409fd8d9dcce0aa1a5656
time="2021-06-22T17:00:12Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:12Z" level=debug msg="Compressing blob on the fly"
time="2021-06-22T17:00:12Z" level=debug msg="Uploading /v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:12Z" level=debug msg="POST https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:13Z" level=debug msg="... not present"
time="2021-06-22T17:00:13Z" level=debug msg="No compression detected"
Copying blob sha256:69d13cbd148f8ce0f721ecfc21192cd7e5ca75fab2184fac7a90a3aa151cd8e4
time="2021-06-22T17:00:13Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:13Z" level=debug msg="Compressing blob on the fly"
time="2021-06-22T17:00:13Z" level=debug msg="Uploading /v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:13Z" level=debug msg="POST https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:13Z" level=debug msg="PATCH https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/ec3bf19b-5a48-4407-a189-7880aa8c8048"
time="2021-06-22T17:00:13Z" level=debug msg="PATCH https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/0edf93c5-aea9-4deb-af5f-31b5720fdc90"
time="2021-06-22T17:00:13Z" level=debug msg="PATCH https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/42861b16-cc30-4565-8fb5-59e6b7bb6a6c"
time="2021-06-22T17:00:13Z" level=debug msg="PATCH https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/d2a9e3eb-6c24-40b5-9470-f8e63e2f9853"
time="2021-06-22T17:00:13Z" level=debug msg="PATCH https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/f4dd87e6-bf6c-4cbe-89c9-e3fccbed75e9"
time="2021-06-22T17:00:13Z" level=debug msg="PATCH https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/6aa8b5e5-a9c9-4af1-bea5-1564429d22c3"
time="2021-06-22T17:00:13Z" level=debug msg="PUT https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/ec3bf19b-5a48-4407-a189-7880aa8c8048?digest=sha256%3A06f696127abebdbd4d7f50c869214a14ac1ab0a5ad836e5b11bd7f6b3ed23293"
time="2021-06-22T17:00:14Z" level=debug msg="Upload of layer sha256:06f696127abebdbd4d7f50c869214a14ac1ab0a5ad836e5b11bd7f6b3ed23293 complete"
time="2021-06-22T17:00:14Z" level=debug msg="PUT https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/6aa8b5e5-a9c9-4af1-bea5-1564429d22c3?digest=sha256%3Ace12853b59f49dd81d63b2a3c3362af6a7f284f0b16accd2194d0ae28dd76b82"
time="2021-06-22T17:00:14Z" level=debug msg="PUT https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/f4dd87e6-bf6c-4cbe-89c9-e3fccbed75e9?digest=sha256%3A5e8dcaf7687f5c9cf34b3c213590668c985242da53291b4131fd85e8a4d4c421"
time="2021-06-22T17:00:14Z" level=debug msg="PUT https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/42861b16-cc30-4565-8fb5-59e6b7bb6a6c?digest=sha256%3A15d2f5b3eaa2cbcde1db1bc8e99e2d777e6d19714e64e0e768f2dc19f157eede"
time="2021-06-22T17:00:14Z" level=debug msg="PUT https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/0edf93c5-aea9-4deb-af5f-31b5720fdc90?digest=sha256%3A044c1498d2c73ed0dfb934a18ca2d3575f4a7ac09dd171173b990800e7faa3cc"
time="2021-06-22T17:00:14Z" level=debug msg="Upload of layer sha256:ce12853b59f49dd81d63b2a3c3362af6a7f284f0b16accd2194d0ae28dd76b82 complete"
time="2021-06-22T17:00:14Z" level=debug msg="Upload of layer sha256:15d2f5b3eaa2cbcde1db1bc8e99e2d777e6d19714e64e0e768f2dc19f157eede complete"
time="2021-06-22T17:00:14Z" level=debug msg="Upload of layer sha256:5e8dcaf7687f5c9cf34b3c213590668c985242da53291b4131fd85e8a4d4c421 complete"
time="2021-06-22T17:00:14Z" level=debug msg="Upload of layer sha256:044c1498d2c73ed0dfb934a18ca2d3575f4a7ac09dd171173b990800e7faa3cc complete"
time="2021-06-22T17:00:15Z" level=debug msg="PUT https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/d2a9e3eb-6c24-40b5-9470-f8e63e2f9853?digest=sha256%3A1850fe5b1c679d9e5b5557cafe9843c3ce251183e901fc4c169608aab29a3259"
time="2021-06-22T17:00:15Z" level=debug msg="Upload of layer sha256:1850fe5b1c679d9e5b5557cafe9843c3ce251183e901fc4c169608aab29a3259 complete"
Copying config sha256:47d8cc97b2c309403ac13968d50706e14daa7e89868f3cb391640d1ac38cc729
time="2021-06-22T17:00:15Z" level=debug msg="No compression detected"
time="2021-06-22T17:00:15Z" level=debug msg="Using original blob without modification"
time="2021-06-22T17:00:15Z" level=debug msg="Checking /v2/microservice-ubi-micro/blobs/sha256:47d8cc97b2c309403ac13968d50706e14daa7e89868f3cb391640d1ac38cc729"
time="2021-06-22T17:00:15Z" level=debug msg="HEAD https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/sha256:47d8cc97b2c309403ac13968d50706e14daa7e89868f3cb391640d1ac38cc729"
time="2021-06-22T17:00:16Z" level=debug msg="... not present"
time="2021-06-22T17:00:16Z" level=debug msg="Uploading /v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:16Z" level=debug msg="POST https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/"
time="2021-06-22T17:00:16Z" level=debug msg="PATCH https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/ea3e0e1f-fe18-4b8b-8586-6846bf914e2b"
time="2021-06-22T17:00:17Z" level=debug msg="PUT https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/blobs/uploads/ea3e0e1f-fe18-4b8b-8586-6846bf914e2b?digest=sha256%3A47d8cc97b2c309403ac13968d50706e14daa7e89868f3cb391640d1ac38cc729"
time="2021-06-22T17:00:17Z" level=debug msg="Upload of layer sha256:47d8cc97b2c309403ac13968d50706e14daa7e89868f3cb391640d1ac38cc729 complete"
Writing manifest to image destination
time="2021-06-22T17:00:17Z" level=debug msg="PUT https://sc-shared-services-team-dev-docker-local.artifactory.swg-devops.com/v2/microservice-ubi-micro/manifests/20210622-165754"
Signing manifest
Storing signatures
time="2021-06-22T17:00:19Z" level=debug msg="Writing to /tmp/sigstore/microservice-ubi-micro@sha256=666ed1e38ada4a2db7f13dae81556042dc4e8d686dd2f0eec448822cb26015f4/signature-1"
time="2021-06-22T17:00:19Z" level=debug msg="Deleting /tmp/sigstore/microservice-ubi-micro@sha256=666ed1e38ada4a2db7f13dae81556042dc4e8d686dd2f0eec448822cb26015f4/signature-2"

Thanks, so this is the direct “blob already exists at destination” path with no BlobInfoCache involvement.

Hum, so runs hard against the fact that the uncompressed MIME type is not specified as valid in schema2 manifests, and at least Quay.io validates that. E.g. using a digest reference to ensure the uncompressed version

% bin/skopeo docker-archive:foo.tar docker://quay.io/mitr/test-archive@sha256:881940c6a398ffeb50f36cb9ce070109e8c069eb67b6adaafa0de890df697d51
… (successfully uploads the uncompressed data)
FATA[0097] Error writing manifest "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":1316,\"digest\":\"sha256:abec9a7a7dc640768e6dc51b6a5728e470411615c62e9ff46215206bde816772\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":183715840,\"digest\":\"sha256:9e6713d530bf59dd0ce8155e1a48372e2ad1773be06a8087deafeb5ad0fed586\"}]}": Error uploading manifest sha256:881940c6a398ffeb50f36cb9ce070109e8c069eb67b6adaafa0de890df697d51 to quay.io/mitr/test-archive: manifest invalid: manifest invalid

(where the digest is the digest of the artificial manifest, obtained by skopeo copy docker-archive:foo.tar dir:x and digesting x/maniest.json)

and afterwards

% bin/skopeo copy docker-archive:foo.tar docker://quay.io/mitr/test-archive:after
…
DEBU[0000] Checking /v2/mitr/test-archive/blobs/sha256:9e6713d530bf59dd0ce8155e1a48372e2ad1773be06a8087deafeb5ad0fed586 
DEBU[0001] ... already exists                           
Writing manifest to image destination
DEBU[0002] PUT https://quay.io/v2/mitr/test-archive/manifests/after 
DEBU[0002] Writing manifest using preferred type application/vnd.docker.distribution.manifest.v2+json failed: Error writing manifest "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":1316,\"digest\":\"sha256:abec9a7a7dc640768e6dc51b6a5728e470411615c62e9ff46215206bde816772\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":183715840,\"digest\":\"sha256:9e6713d530bf59dd0ce8155e1a48372e2ad1773be06a8087deafeb5ad0fed586\"}]}": Error uploading manifest after to quay.io/mitr/test-archive: manifest invalid: manifest invalid 
DEBU[0002] Trying to use manifest type application/vnd.docker.distribution.manifest.v1+prettyjws… 
Writing manifest to image destination
DEBU[0002] PUT https://quay.io/v2/mitr/test-archive/manifests/after 
…

i.e. Quay.io rejects the uncompressed value, and that causes a fallback to schema1.

On balance, it seems distinctly more useful to use schema2 with an incorrect MIME type (which is, in practice, quite interoperable) than to fall back back to schema1, where the digest changes with each registry/repository change.

OTOH it’s extremely unclean that we currently achieve the more useful result by such an indirect approach.

The fundamental difficulty is that we need to express a Docker schema2 image (because the config in docker/internal/tarfile is a schema2 config), but with uncompressed layers — which just doesn’t exist in the spec; but the rest of the copy code makes a presumption that the input representation is the ~preferred one and should not be changed frivolously, and that layer reuse is always better than layer upload.

Implementation artifacts involved:

We create a schema2 manifest for docker/internal/tarfile contents, instead of representing the (multi-image???) tarball manifest format directly (well, there are two manifest formats, and we want to at least write both…)
The code first decides whether/how to write layers to the destination, and then decides on the manifest format (i.e. which layer formats are even valid). This is even more pronounced with Zstd compression, which is OCI-only.
(We actually want skopeo copy docker-archive:… dir:… to save the artificial schema2 manifest with uncompressed layers, to correctly represent internal state. How does that differ from the registry upload situation?)
(Quay.io validates the manifest enough to reject uncompressed MIME types, but not enough to reject uncompressed data with compressed MIME types… lucky for us.)

I can’t think of a simple fix. We’ll probably in some vague future have to teach the copy code about manifest-format-dependent layer formats, but that’s non-trivial work and it has serious downsides as well — if we upload the layer in one format, then try uploading the manifest, and that fails, deleting blobs might not even be possible.

I’ll keep this open for how, hoping that someone else can come up with a good solution, or just to keep this in mind for the future.

@mtrmac Looks like there is no easy solution. But are there any workarounds to force it to upload the compressed layer or have the manifest be correct?

Depending on the registry, pushing to a different repository (where the uncompressed layer does not yet exist) might work — or it might find it anyway; an inter-repo copy would then be easy.

Or use an intermediate skopeo copy --dest-compress $source dir:$tmp, and then copy from dir:$tmp.

Would it help to do skopeo copy --debug --retry-times 3 --dest-creds ****:**** --sign-by <redacted> docker-daemon:<local_image> containers-storage://<image> and then copy (push) to registry? that is, would a stopover in containers-storage suffice to get the layers metadata cleaned up before push to registry?

Yes, to the extent that the MIME type will be “uncompressed” (but Quay.io will reject that and cause a downgrade to schema1.) But the uncompressed version will be preferred if it already exists on the registry, if the process is not already using containers-storage, this will cause creating individual files on the filesystem, with all the related seeks and cache flushes, and probably be much more expensive than the dir: intermediate.

… but that’s a fair point; if c/storage uses the uncompressed manifests, it’s more attractive for the tarfile transports (docker-archive: and docker-daemon:) to do the same, regardless of the Quay impact.

After https://github.com/containers/image/pull/2068 , users should be able to force layer compression instead of reuse of uncompressed layers.

The underlying design issue remains outstanding.

containers / image

docker/internal/tarfile has incorrect media type in manifests; copy layer format is not manifest-format dependent #1270