search

CircleCI-Public / cimg-aws

The CircleCI AWS Docker Convenience Image.
https://circleci.com/developer/images/image/cimg/aws
MIT License
2 stars 2 forks source link

Bug Report: Job using cimg/aws:2023.07.1 for ARM64 cannot run steps #19

Closed kelvintaywl closed 1 year ago

kelvintaywl commented 1 year ago

Describe the bug A clear and concise description of what the bug is.

When using the cimg/aws:2023.07.1 image for a Docker executor job (& specifying for ARM), the job failed with no steps run. See failed job here: https://app.circleci.com/pipelines/github/kelvintaywl-cci/open-arms/18/workflows/5afa415c-929f-4612-acd3-2c57c9c8296d/jobs/45

To Reproduce

Here is the minimal config to reproduce:

version: 2.1

jobs:
  build:
    docker:
      - image: cimg/aws:2023.07.1
    resource_class: arm.large
    steps:
      - run: |
          uname -m

Expected behavior

It should run as with other supported ARM-available images like cimg/base:current : https://app.circleci.com/pipelines/github/kelvintaywl-cci/open-arms/18/workflows/5afa415c-929f-4612-acd3-2c57c9c8296d/jobs/47

Workarounds

Are there any current workarounds for this bug that can be used currently?

Not that I am aware of yet!

Screenshots and Build Links If possible, add screenshots and links to jobs to help explain your problem.

Additional context

I observed that the other "main" images listed on our Discuss ran fine. (related workflow here)

kelvintaywl commented 1 year ago

Hi folks,

in case this can help narrow down the cause:

# NOTE: on my local Mac (M1)

# confirm my local is on ARM64
$ uname -a
Darwin [redacted] 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103 arm64

$ docker version -f "{{ .Server }}"
{{Docker Desktop 4.21.1 (114176)} [{Engine 24.0.2 map[ApiVersion:1.43 Arch:arm64 BuildTime:Thu May 25 21:50:59 2023 Experimental:false GitCommit:659604f GoVersion:go1.20.4 KernelVersion:5.15.49-linuxkit-pr MinAPIVersion:1.12 Os:linux]} {containerd 1.6.21 map[GitCommit:3dce8eb055cbb6872793272b4f20ed16117344f8]} {runc 1.1.7 map[GitCommit:v1.1.7-0-g860f061]} {docker-init 0.19.0 map[GitCommit:de40ad0]}] 24.0.2 1.43 1.12 659604f go1.20.4 linux arm64 5.15.49-linuxkit-pr false 2023-05-25T21:50:59.000000000+00:00}

# confirm my locally-pulled cimg/aws:2023.07 image is for linux/arm64
$ docker inspect cimg/aws:2023.07 | jq '.[0].Os + "/" + .[0].Architecture'
"linux/arm64"

# Interestingly, the image seems to be amd64-based still...
$ docker run --rm cimg/aws:2023.07 uname -a
Linux e23907c9414d 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

# NOTE: compared to my other images....
$ docker run --rm cimg/base:current uname -a
Linux ee9482f34d07 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

$ docker run --rm cimg/deploy:2023.07 uname -a
Linux 57a534cd62e8 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

# NOTE: cimg/azure:2023.07 is also seeing similar issues...
$ docker run --rm cimg/azure:2023.07 uname -a
Linux f82e02f589c3 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
kelvintaywl commented 1 year ago

From what I can find:

[hypothesis]
I wonder if the issue is because the cimg/deploy:2023:07-node base image (arm64 variant) was not available at least when we built the arm64 variants for cimg/aws:2023.07 & cimg/azure:2023.07 -> these 2 images end up being based on a amd64 variant instead.

[proof] I can see the cimg/aws:2023.07 arm64 variant was pulling the cimg/deploy:2023.07-node of sha256@ 8276af005bbf57be...: https://app.circleci.com/pipelines/github/CircleCI-Public/cimg-aws/45/workflows/fba37017-d030-4d9a-9607-c740920dd278/jobs/48?invite=true#step-104-1578

This was the latest image for the cimg/deploy:2023.07-node image tag then. If we pull this image down, and inspect, we see this is for amd64 not arm64:

# NOTE: on my local Mac M1
$ docker run --rm cimg/deploy@sha256:8276af005bbf57beeffd5a09b94065f87e083993c5716f0008c11383a28db644 uname -a
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Linux de92c3fac04f 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ docker inspect cimg/deploy@sha256:8276af005bbf57beeffd5a09b94065f87e083993c5716f0008c11383a28db644 | jq '.[0].Os + "/" + .[0].Architecture'
"linux/amd64"
BytesGuy commented 1 year ago

This is verified as resolved in the cimg/aws:2023.08.1 release