search

containerd / containerd

An open and reliable container runtime
https://containerd.io
Apache License 2.0
16.99k stars 3.37k forks source link

Cannot pull containers from mcr.microsoft.com due to platform matching #7439

Open krotz-dieter opened 1 year ago

krotz-dieter commented 1 year ago

Description

On Windows 11 Version 10.0.22000.978

docker pull mcr.microsoft.com/windows/nanoserver:ltsc2022

works fine, it pulls the image and docker run --isolation process -m 4G -it --rm --name test-run mcr.microsoft.com/windows/nanoserver:ltsc2022 also works ! With containerd

crictl pull mcr.microsoft.com/windows/nanoserver:ltsc2022

we get the error:

E0927 11:53:53.142414   23840 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"mcr.microsoft.com/windows/nanoserver:ltsc2022\": no match for platform in manifest: not found" image="mcr.microsoft.com/windows/nanoserver:ltsc2022"
time="2022-09-27T11:53:53+02:00" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \"mcr.microsoft.com/windows/nanoserver:ltsc2022\": no match for platform in manifest: not found"

The logs from containerd service:

time="2022-09-27T11:29:29.107877800+02:00" level=info msg="PullImage \"mcr.microsoft.com/windows/nanoserver:ltsc2022\""
time="2022-09-27T11:29:29.114833800+02:00" level=debug msg=resolving host=mcr.microsoft.com
time="2022-09-27T11:29:29.114833800+02:00" level=debug msg="do request" host=mcr.microsoft.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.6.8 request.method=HEAD url="https://mcr.microsoft.com/v2/windows/nanoserver/manifests/ltsc2022"
time="2022-09-27T11:29:29.285758600+02:00" level=debug msg="fetch response received" host=mcr.microsoft.com response.header.access-control-expose-headers=Docker-Content-Digest response.header.access-control-expose-headers.1=WWW-Authenticate response.header.access-control-expose-headers.2=Link response.header.access-control-expose-headers.3=X-Ms-Correlation-Request-Id response.header.content-length=350 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Tue, 27 Sep 2022 09:29:37 GMT" response.header.docker-content-digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755\"" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.strict-transport-security.1="max-age=31536000; includeSubDomains" response.header.x-cache=CONFIG_NOCACHE response.header.x-content-type-options=nosniff response.header.x-mcr-env=prod response.header.x-mcr-privacy="https://privacy.microsoft.com/en-us/privacystatement" response.header.x-ms-client-request-id= response.header.x-ms-correlation-request-id=7288f4f6-8a73-404f-aac0-3b2729856453 response.header.x-ms-request-id=dabae9ae-dcad-4a0c-88bd-46fc12c2b699 response.header.x-msedge-ref="Ref A: F822B534419B485F97D8620C44124705 Ref B: MUC30EDGE0415 Ref C: 2022-09-27T09:29:38Z" response.status="200 OK" url="https://mcr.microsoft.com/v2/windows/nanoserver/manifests/ltsc2022"
time="2022-09-27T11:29:29.285758600+02:00" level=debug msg=resolved desc.digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" host=mcr.microsoft.com
time="2022-09-27T11:29:29.285758600+02:00" level=debug msg=fetch digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=350
time="2022-09-27T11:29:29.293765100+02:00" level=debug msg="do request" digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=containerd/v1.6.8 request.method=GET size=350 url="https://mcr.microsoft.com/v2/windows/nanoserver/manifests/sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755"
time="2022-09-27T11:29:29.464871400+02:00" level=debug msg="fetch response received" digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.access-control-expose-headers=Docker-Content-Digest response.header.access-control-expose-headers.1=WWW-Authenticate response.header.access-control-expose-headers.2=Link response.header.access-control-expose-headers.3=X-Ms-Correlation-Request-Id response.header.content-length=350 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Tue, 27 Sep 2022 09:29:38 GMT" response.header.docker-content-digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755\"" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.strict-transport-security.1="max-age=31536000; includeSubDomains" response.header.x-cache=CONFIG_NOCACHE response.header.x-content-type-options=nosniff response.header.x-mcr-env=prod response.header.x-mcr-privacy="https://privacy.microsoft.com/en-us/privacystatement" response.header.x-ms-client-request-id= response.header.x-ms-correlation-request-id=107253db-1b65-4fd5-91c8-56324ae43208 response.header.x-ms-request-id=7ac3b319-bedf-466c-ae64-8512bac5c0b3 response.header.x-msedge-ref="Ref A: A03DFEAA8D1C446E92EA49CFEAF58449 Ref B: MUC30EDGE0215 Ref C: 2022-09-27T09:29:38Z" response.status="200 OK" size=350 url="https://mcr.microsoft.com/v2/windows/nanoserver/manifests/sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755"
time="2022-09-27T11:29:29.475871600+02:00" level=error msg="PullImage \"mcr.microsoft.com/windows/nanoserver:ltsc2022\" failed" error="rpc error: code = NotFound desc = failed to pull and unpack image \"mcr.microsoft.com/windows/nanoserver:ltsc2022\": no match for platform in manifest: not found"
time="2022-09-27T11:29:29.544241300+02:00" level=debug msg="remove content" key="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755"
time="2022-09-27T11:29:29.546883800+02:00" level=debug msg="schedule content cleanup"
time="2022-09-27T11:29:29.547921900+02:00" level=debug msg="removed content" digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755"
time="2022-09-27T11:29:29.547921900+02:00" level=debug msg="content garbage collected" d=1.0381ms
time="2022-09-27T11:29:29.548439700+02:00" level=debug msg="garbage collected" d=3.1736ms

The expectation would be that it would work also within containerd.

Steps to reproduce the issue

crictl pull mcr.microsoft.com/windows/nanoserver:ltsc2022

Describe the results you received and expected

The expectation woould be that it would work also within containerd.

What version of containerd are you using?

containerd github.com/containerd/containerd v1.6.8 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6

Any other relevant information

crictl info

{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "C:\\k\\cni\\bin"
    ],
    "PluginConfDir": "C:\\k\\containerd\\cni\\conf",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "cbr0",
          "CNIVersion": "0.2.0",
          "Plugins": [
            {
              "Network": {
                "cniVersion": "0.2.0",
                "name": "cbr0",
                "type": "flannel",
                "capabilities": {
                  "dnsCapabilities": true,
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"dnsCapabilities\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"delegate\":{\"AdditionalArgs\":[{\"Name\":\"EndpointPolicy\",\"Value\":{\"Settings\":{\"Exceptions\":[\"172.20.0.0/16\",\"172.21.0.0/16\",\"172.19.1.0/24\",\"192.168.44.0/24\"]},\"Type\":\"OutBoundNAT\"}}],\"dns\":{\"Nameservers\":[\"172.20.1.2\"]},\"optionalFlags\":{\"forceBridgeGateway\":true,\"loopbackDSR\":false},\"type\":\"sdnbridge\"},\"name\":\"cbr0\",\"type\":\"flannel\"}"
            }
          ],
          "Source": "{\"cniVersion\":\"0.2.0\",\"name\":\"cbr0\",\"plugins\":[{\"capabilities\":{\"dnsCapabilities\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"delegate\":{\"AdditionalArgs\":[{\"Name\":\"EndpointPolicy\",\"Value\":{\"Settings\":{\"Exceptions\":[\"172.20.0.0/16\",\"172.21.0.0/16\",\"172.19.1.0/24\",\"192.168.44.0/24\"]},\"Type\":\"OutBoundNAT\"}}],\"dns\":{\"Nameservers\":[\"172.20.1.2\"]},\"optionalFlags\":{\"forceBridgeGateway\":true,\"loopbackDSR\":false},\"type\":\"sdnbridge\"},\"name\":\"cbr0\",\"type\":\"flannel\"}]}"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "windows",
      "defaultRuntimeName": "runhcs-wcow-process",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "runtimes": {
        "runhcs-wcow-process": {
          "runtimeType": "io.containerd.runhcs.v1",
          "runtimePath": "",
          "runtimeEngine": "",
          "PodAnnotations": [],
          "ContainerAnnotations": [],
          "runtimeRoot": "",
          "options": {},
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": "",
          "cniConfDir": "",
          "cniMaxConfNum": 0
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": false,
      "discardUnpackedLayers": false,
      "ignoreRdtNotEnabledErrors": false
    },
    "cni": {
      "binDir": "C:\\k\\cni\\bin",
      "confDir": "C:\\k\\containerd\\cni\\conf",
      "maxConfNum": 1,
      "confTemplate": "",
      "ipPref": ""
    },
    "registry": {
      "configPath": "",
      "mirrors": {},
      "configs": {},
      "auths": {},
      "headers": {
        "User-Agent": [
          "containerd/v1.6.8"
        ]
      }
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 0,
    "sandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.6",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": false,
    "disableHugetlbController": false,
    "device_ownership_from_security_context": false,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "enableUnprivilegedPorts": false,
    "enableUnprivilegedICMP": false,
    "containerdRootDir": "C:\\containerd\\root",
    "containerdEndpoint": "\\\\.\\pipe\\containerd-containerd",
    "rootDir": "C:\\containerd\\root\\io.containerd.grpc.v1.cri",
    "stateDir": "C:\\containerd\\state\\io.containerd.grpc.v1.cri"
  },
  "golang": "go1.17.13",
  "lastCNILoadStatus": "OK",
  "lastCNILoadStatus.default": "OK"
}

Show configuration if it is related to CRI plugin.

disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "C:\\containerd\\root"
state = "C:\\containerd\\state"
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "\\\\.\\pipe\\containerd-containerd"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = false
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "mcr.microsoft.com/oss/kubernetes/pause:3.6"
    selinux_category_range = 0
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = false
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "C:\\k\\cni\\bin"
      conf_dir = "C:\\k\\containerd\\cni\\conf"
      conf_template = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runhcs-wcow-process"
      disable_snapshot_annotations = false
      discard_unpacked_layers = false
      no_pivot = false
      snapshotter = "windows"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process]
          base_runtime_spec = ""
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runhcs.v1"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "C:\\k\\containerd\\root\\opt"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["windows/amd64", "linux/amd64"]

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["windows", "windows-lcow"]

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "C:\\k\\containerd\\ocicrypt\\keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=C:\\k\\containerd\\ocicrypt\\ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "C:\\k\\containerd\\ocicrypt\\keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=C:\\k\\containerd\\ocicrypt\\ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0
estesp commented 1 year ago

Looks like a platform matching issue with your Windows version and the data in the image? @dcantah or @jterry75 ?

krotz-dieter commented 1 year ago

Why is it working then with the docker daemon ? The container is the same: mcr.microsoft.com/windows/nanoserver:ltsc2022, it should behave the same with containerd also.

estesp commented 1 year ago

The Docker daemon has its own implementation of the registry client interactions; our platform matching code is probably different because of this. It's possible we have a bug or tighter restriction than Docker engine on the Windows matching; we will have to dig into that and understand why; that's why I copied in our Windows experts :)

dcantah commented 1 year ago

Yea Phil nailed it, the platform matcher for CRI (k8s entrypoint) matches to the hosts build exactly and will reject anything else. So in this case 20348 != 22000 (win11). Reason for the restriction being hyper-v containers weren't supported until very recently through the CRI plugin here, so pulling any other image wasn't valid as windows used to mandate the host <-> container image build to match. That was actually relaxed recently, so the ltsc2022 image should work fine on any host that's higher in build number, so we should probably put in some logic for this as what you're trying to do is completely valid.

krotz-dieter commented 1 year ago

Thanks for the info, can you point me to the code of the platform matcher CRI ? Would like to try out myself to see if I can get it running. Also, do you plan to incorporate such a change for Windows containers ?

dcantah commented 1 year ago

@SyngoPredevelopment Sure, https://github.com/containerd/containerd/blob/main/platforms/defaults_windows.go. And yes! Need to work out any edge cases but yes, it'd be a shame to not allow this functionality if it's supported in the platform now

krotz-dieter commented 1 year ago

OK, many thanks

fuweid commented 1 year ago

@dcantah @SyngoPredevelopment any update here?

dcantah commented 1 year ago

Hey, no updates on my end but https://github.com/containerd/containerd/pull/7856 looks to address platform matching discrepancies here a bit

krotz-dieter commented 1 year ago

We solved it in another way, we add in the manifest the supported OS versions.

weipah commented 1 year ago

Solution is cooking there, I guess: #8137

Dirty workaround is to match the os.version in manifest file to your client os version.