squat / generic-device-plugin

A Kubernetes device plugin to schedule generic Linux devices
Apache License 2.0
208 stars 23 forks source link

Image platform tag is incorrect for arm and arm64 images #23

Open jpflouret opened 1 year ago

jpflouret commented 1 year ago

The os/arch label is wrong on the docker images for both arm and arm64. Docker hub has linux/amd64 for all 3 different digests in the latest tag.

image

Running the image throught docker on an arm64 machine to verify I get this warning:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
squat commented 1 year ago

Oh wow, thanks for catching this!! This seems to be a regression that was introduced sometime after bd0d5d18081e0b56b00271688f2ded15e6a1b3c3.

Interestingly, the GitHub registry images are correctly labeled with the right architectures. Until this bug in CI is fixed, I would suggest using the GHCR images rather than the Docker Hub images.

squat commented 1 year ago

doing some more inspection, there seem to be some funny issues with Docker Hub in particular. When inspecting the manifest list for the container, it seems that the metadata correctly lists the architectures for the images:

docker manifest inspect squat/generic-device-plugin
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 528,
         "digest": "sha256:93cba71275c1a2c2bd443b72c9593edfd703859b355520db4a03e3a9b548c2f4",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 528,
         "digest": "sha256:046b0027a11d40836349c685727aac435a9418ea6771707e937f931acdd9641c",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 528,
         "digest": "sha256:4c4e3e29c7827c0d3a99216fa9eab468d7c194d45d97fa0eb8bdf4d14bcf6b2a",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      }
   ]
}

However, interestingly the Docker UI does not show the correct platforms.

squat commented 1 year ago

This issue is caused by the fact that the metadata for the underlying images that make up the manifest list do not explicitly set a platform at build time, which causes their label to default to amd64. Although the images really are compiled for non-amd64 architectures and the manifest list is correctly labeled, the UI and the Docker tool will show the warning. The images should still run correctly on arm/arm64.

I'm investigating the easiest way to enhance CI to label the underlying images with the correct platform.

jpflouret commented 1 year ago

Have you considered using docker buildx plugin? You could build and tag the images for multiple architectures and also push the image (with multiple tags) to docker hub and ghcr all in a single command.

zestysoft commented 1 year ago

FWIW, I deployed this in kubernetes across arm and amd nodes and had no trouble with platform incompatibilities.