build fails on master and v2.4.1 on arm64

liyimeng commented 1 year ago

Kairos version:

master/v2.4.1

CPU architecture, OS, and Version:

ARM64

Describe the bug

I try to build kairos on master/v2.4.1 branch, it fails at building the framework. Not sure what I have been missing To Reproduce

checkout master branch
run earthy.sh +iso --FLAVOR=alpine
wait for a while and seeing a failure.

Expected behavior

build should go OK

Logs


./earthly.sh   +iso --FLAVOR=alpine 
detected cgroups v2; earthly-entrypoint.sh running under pid=1 with controllers "cpuset cpu io memory hugetlb pids rdma" in group 0::/
loading config values from "/etc/.earthly/config.yml"
       buildkitd | Starting buildkit daemon as a docker container (earthly-buildkitd)...
       bootstrap | Warning: Bootstrapping buildkit failed: maybe start buildkitd: start: could not start buildkit: 1 error occurred:
       bootstrap |  * command failed: docker run --privileged --env BUILDKIT_MAX_PARALLELISM=20 --env BUILDKIT_DEBUG=false --env BUILDKIT_TCP_TRANSPORT_ENABLED=true --env BUILDKIT_TLS_ENABLED=true --label dev.earthly.settingshash=53bb7043424f0e87 --mount type=volume,source=earthly-cache,dst=/tmp/earthly --mount type=bind,source=/root/.earthly/certs/ca_cert.pem,dst=/etc/ca.pem,ro=true --mount type=bind,source=/root/.earthly/certs/buildkit_cert.pem,dst=/etc/cert.pem,ro=true --mount type=bind,source=/root/.earthly/certs/buildkit_key.pem,dst=/etc/key.pem,ro=true --publish 127.0.0.1:8371:8371/tcp --publish 127.0.0.1:8372:8372/tcp -d --pull missing --name earthly-buildkitd docker.io/earthly/buildkitd:v0.7.20: exit status 125: docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /root/.earthly/certs/ca_cert.pem.
       bootstrap | See 'docker run --help'.: exit status 125
       bootstrap | 
       bootstrap | Bootstrapping successful.
loading config values from "/etc/.earthly/config.yml"
Init 🚀
————————————————————————————————————————————————————————————————————————————————

       buildkitd | Connecting to tcp://5062f758fc6f:8372...
       buildkitd | ...Done
       buildkitd | Version github.com/earthly/buildkit v0.7.20 af796f0cb8ff6cf53fdebd543ff1e7cd173c00a1
       buildkitd | Platforms: linux/arm64 (native) linux/amd64 linux/amd64/v2 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/arm/v7 linux/arm/v6
       buildkitd | Utilization: 0 other builds, 0/20 op load
       buildkitd | GC stats: 0 B cache, avg GC duration 0s, all-time GC duration 0s, last GC duration 0s, last cleared 0 B

Build 🔧
————————————————————————————————————————————————————————————————————————————————

          alpine | --> Load metadata alpine linux/arm64
           +base | FLAVOR=alpine
           +base | --> FROM alpine
           +base | [----------] 100% FROM alpine
           +base | FLAVOR=alpine
           +base | *cached* --> mkdir /run
           +base | FLAVOR=alpine
           +base | *cached* --> IF [ "$FLAVOR" = "ubuntu" ]
        +version | FLAVOR=alpine
        +version | *cached* --> RUN apk add git
        +version | FLAVOR=alpine
        +version | --> COPY . ./
        +version | FLAVOR=alpine
        +version | --> mkdir /run/buildargs
        +version | FLAVOR=alpine
        +version | --> ARG _GIT_VERSION = RUN $(git describe --always --tags --dirty)
        +version | FLAVOR=alpine
        +version | --> mkdir /run
        +version | FLAVOR=alpine
        +version | --> IF [ "$K3S_VERSION" != "" ]
        +version | FLAVOR=alpine
        +version | --> RUN --no-cache echo ${_GIT_VERSION}${_K3S_VERSION} > VERSION
        +version | FLAVOR=alpine
        +version | --> mkdir /run/buildargs
        +version | FLAVOR=alpine
        +version | --> ARG VERSION = RUN $(cat VERSION)
        +version | FLAVOR=alpine
        +version | *cached* --> SAVE ARTIFACT VERSION +version/VERSION
            +iso | FLAVOR=alpine
            +iso | *cached* --> COPY +version/VERSION ./
            +iso | FLAVOR=alpine
            +iso | *cached* --> mkdir /run/buildargs
            +iso | FLAVOR=alpine
            +iso | *cached* --> ARG VERSION = RUN $(cat VERSION)
q/k/osbuilder-tools:v0.8.6 | --> Load metadata quay.io/kairos/osbuilder-tools:v0.8.6 linux/arm64
     +base-image | FLAVOR=alpine
     +base-image | *cached* --> IF [ "$BASE_IMAGE" = "" ]
     +base-image | FLAVOR=alpine
     +base-image | *cached* --> IF [[ "$FLAVOR" =~ ^alpine* ]]
     +base-image | FLAVOR=alpine
     +base-image | *cached* --> mkdir /run/buildargs
     +base-image | FLAVOR=alpine
     +base-image | *cached* --> ARG SIMPLE_FLAVOR = RUN $(echo alpine | sed s/-arm-.*//)
   golang:alpine | --> Load metadata golang:alpine linux/arm64
q/luet/base:0.35.0 | --> Load metadata quay.io/luet/base:0.35.0 linux/arm64
           +luet | FLAVOR=alpine
           +luet | --> FROM quay.io/luet/base:0.35.0
      +framework | FLAVOR=alpine
      +framework | --> FROM golang:alpine
           +luet | [----------] 100% FROM quay.io/luet/base:0.35.0
           +luet | FLAVOR=alpine
           +luet | *cached* --> SAVE ARTIFACT /usr/bin/luet +luet/luet
      +framework | [----------] 100% FROM golang:alpine
      +framework | FLAVOR=alpine
      +framework | *cached* --> WORKDIR /build
      +framework | FLAVOR=alpine
      +framework | *cached* --> COPY ./profile-build /build
      +framework | FLAVOR=alpine
      +framework | *cached* --> COPY framework-profile.yaml /build
      +framework | FLAVOR=alpine
      +framework | *cached* --> COPY +luet/luet /usr/bin/luet
      +framework | FLAVOR=alpine
      +framework | --> RUN go run main.go ${FLAVOR} framework-profile.yaml /framework
      +framework | go: downloading github.com/kairos-io/kairos-sdk v0.0.14
      +framework | go: downloading github.com/urfave/cli v1.22.14
      +framework | go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.2
      +framework | go: downloading gopkg.in/yaml.v3 v3.0.1
      +framework | go: downloading github.com/containerd/containerd v1.7.1
      +framework | go: downloading github.com/denisbrodbeck/machineid v1.0.1
      +framework | go: downloading github.com/google/go-containerregistry v0.15.2
      +framework | go: downloading github.com/hashicorp/go-multierror v1.1.1
      +framework | go: downloading github.com/qeesung/image2ascii v1.0.1
      +framework | go: downloading github.com/joho/godotenv v1.5.1
      +framework | go: downloading github.com/pterm/pterm v0.12.63
      +framework | go: downloading gopkg.in/yaml.v1 v1.0.0-20140924161607-9f9df34309c0
      +framework | go: downloading github.com/russross/blackfriday/v2 v2.1.0
      +framework | go: downloading github.com/containerd/continuity v0.3.0
      +framework | go: downloading golang.org/x/sys v0.10.0
      +framework | go: downloading github.com/docker/cli v23.0.5+incompatible
      +framework | go: downloading github.com/mitchellh/go-homedir v1.1.0
      +framework | go: downloading github.com/docker/docker v23.0.5+incompatible
      +framework | go: downloading github.com/docker/distribution v2.8.1+incompatible
      +framework | go: downloading golang.org/x/sync v0.2.0
      +framework | go: downloading github.com/opencontainers/go-digest v1.0.0
      +framework | go: downloading github.com/hashicorp/errwrap v1.1.0
      +framework | go: downloading atomicgo.dev/cursor v0.1.3
      +framework | go: downloading atomicgo.dev/keyboard v0.2.9
      +framework | go: downloading atomicgo.dev/schedule v0.0.2
      +framework | go: downloading github.com/gookit/color v1.5.3
      +framework | go: downloading github.com/lithammer/fuzzysearch v1.1.8
      +framework | go: downloading github.com/mattn/go-runewidth v0.0.14
      +framework | go: downloading golang.org/x/term v0.10.0
      +framework | go: downloading golang.org/x/text v0.11.0
      +framework | go: downloading github.com/sirupsen/logrus v1.9.0
      +framework | go: downloading github.com/pkg/errors v0.9.1
      +framework | go: downloading github.com/docker/go-connections v0.4.0
      +framework | go: downloading github.com/docker/go-units v0.5.0
      +framework | go: downloading github.com/opencontainers/image-spec v1.1.0-rc3
      +framework | go: downloading github.com/containerd/stargz-snapshotter/estargz v0.14.3
      +framework | go: downloading github.com/containerd/console v1.0.3
      +framework | go: downloading github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e
      +framework | go: downloading github.com/rivo/uniseg v0.4.4
      +framework | go: downloading github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646
      +framework | go: downloading github.com/docker/docker-credential-helpers v0.7.0
      +framework | go: downloading github.com/klauspost/compress v1.16.5
      +framework | go: downloading golang.org/x/net v0.10.0
      +framework | go: downloading github.com/vbatts/tar-split v0.11.3
      +framework | go: downloading github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59
      +framework | go: downloading github.com/mattn/go-isatty v0.0.17
      +framework | go: downloading github.com/wayneashleyberry/terminal-dimensions v1.1.0
      +framework | go: downloading github.com/gogo/protobuf v1.3.2
      +framework | No profile found
      +framework | exit status 1
      +framework | ERROR Earthfile line 261:4
      +framework |       The command
      +framework |           RUN go run main.go ${FLAVOR} framework-profile.yaml /framework
      +framework |       did not complete successfully. Exit code 1

================================== ❌ FAILURE ===================================

      +framework *failed* | Repeating the failure error...
      +framework *failed* | FLAVOR=alpine
      +framework *failed* | --> RUN go run main.go ${FLAVOR} framework-profile.yaml /framework
      +framework *failed* | go: downloading github.com/kairos-io/kairos-sdk v0.0.14
      +framework *failed* | go: downloading github.com/urfave/cli v1.22.14
      +framework *failed* | go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.2
      +framework *failed* | go: downloading gopkg.in/yaml.v3 v3.0.1
      +framework *failed* | go: downloading github.com/containerd/containerd v1.7.1
      +framework *failed* | go: downloading github.com/denisbrodbeck/machineid v1.0.1
      +framework *failed* | go: downloading github.com/google/go-containerregistry v0.15.2
      +framework *failed* | go: downloading github.com/hashicorp/go-multierror v1.1.1
      +framework *failed* | go: downloading github.com/qeesung/image2ascii v1.0.1
      +framework *failed* | go: downloading github.com/joho/godotenv v1.5.1
      +framework *failed* | go: downloading github.com/pterm/pterm v0.12.63
      +framework *failed* | go: downloading gopkg.in/yaml.v1 v1.0.0-20140924161607-9f9df34309c0
      +framework *failed* | go: downloading github.com/russross/blackfriday/v2 v2.1.0
      +framework *failed* | go: downloading github.com/containerd/continuity v0.3.0
      +framework *failed* | go: downloading golang.org/x/sys v0.10.0
      +framework *failed* | go: downloading github.com/docker/cli v23.0.5+incompatible
      +framework *failed* | go: downloading github.com/mitchellh/go-homedir v1.1.0
      +framework *failed* | go: downloading github.com/docker/docker v23.0.5+incompatible
      +framework *failed* | go: downloading github.com/docker/distribution v2.8.1+incompatible
      +framework *failed* | go: downloading golang.org/x/sync v0.2.0
      +framework *failed* | go: downloading github.com/opencontainers/go-digest v1.0.0
      +framework *failed* | go: downloading github.com/hashicorp/errwrap v1.1.0
      +framework *failed* | go: downloading atomicgo.dev/cursor v0.1.3
      +framework *failed* | go: downloading atomicgo.dev/keyboard v0.2.9
      +framework *failed* | go: downloading atomicgo.dev/schedule v0.0.2
      +framework *failed* | go: downloading github.com/gookit/color v1.5.3
      +framework *failed* | go: downloading github.com/lithammer/fuzzysearch v1.1.8
      +framework *failed* | go: downloading github.com/mattn/go-runewidth v0.0.14
      +framework *failed* | go: downloading golang.org/x/term v0.10.0
      +framework *failed* | go: downloading golang.org/x/text v0.11.0
      +framework *failed* | go: downloading github.com/sirupsen/logrus v1.9.0
      +framework *failed* | go: downloading github.com/pkg/errors v0.9.1
      +framework *failed* | go: downloading github.com/docker/go-connections v0.4.0
      +framework *failed* | go: downloading github.com/docker/go-units v0.5.0
      +framework *failed* | go: downloading github.com/opencontainers/image-spec v1.1.0-rc3
      +framework *failed* | go: downloading github.com/containerd/stargz-snapshotter/estargz v0.14.3
      +framework *failed* | go: downloading github.com/containerd/console v1.0.3
      +framework *failed* | go: downloading github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e
      +framework *failed* | go: downloading github.com/rivo/uniseg v0.4.4
      +framework *failed* | go: downloading github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646
      +framework *failed* | go: downloading github.com/docker/docker-credential-helpers v0.7.0
      +framework *failed* | go: downloading github.com/klauspost/compress v1.16.5
      +framework *failed* | go: downloading golang.org/x/net v0.10.0
      +framework *failed* | go: downloading github.com/vbatts/tar-split v0.11.3
      +framework *failed* | go: downloading github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59
      +framework *failed* | go: downloading github.com/mattn/go-isatty v0.0.17
      +framework *failed* | go: downloading github.com/wayneashleyberry/terminal-dimensions v1.1.0
      +framework *failed* | go: downloading github.com/gogo/protobuf v1.3.2
      +framework *failed* | No profile found
      +framework *failed* | exit status 1
      +framework *failed* | ERROR Earthfile line 261:4
      +framework *failed* |       The command
      +framework *failed* |           RUN go run main.go ${FLAVOR} framework-profile.yaml /framework
      +framework *failed* |       did not complete successfully. Exit code 1


**Additional context**
<!-- Add any other context about the problem here. -->

Itxaka commented 1 year ago

Hello @liyimeng

You are setting a random flavor name, so when framework tries to include the packages it tries searc hing for the given flavor in the profiles, but alpine is not a valid falvor.

You should use either alpine-opensuse-leap or alpine-ubuntu as the ---FLAVOR to get an apline iso.

Itxaka commented 1 year ago

You can seea ll supported out of the box flavors to build with earthly under https://github.com/kairos-io/kairos/blob/master/framework-profile.yaml

liyimeng commented 1 year ago

Hello @liyimeng

You are setting a random flavor name, so when framework tries to include the packages it tries searc hing for the given flavor in the profiles, but alpine is not a valid falvor.

You should use either alpine-opensuse-leap or alpine-ubuntu as the ---FLAVOR to get an apline iso.

@Itxaka thanks! In this case, the we probably need to update document at https://kairos.io/docs/development/development/, where it is misleading, what do u think?

liyimeng commented 1 year ago

@Itxaka by the way, even using alpine-ubuntu, it fails as well. since package grub-bios is not available on arm64 in alpine3.18

I re-open this issue, hope it is OK with u

Itxaka commented 1 year ago

Hi @liyimeng ,

yeah, for arm64 isos, we have only tested opensuse-leap as its the only one that works out of the box. For other flavors, the only supported arm64 devices is rpi3/rpi4.

Are you building an generic arm64 iso for an arm machine?

I will send a patch to install grub-bios only under the x86 arch in the alpine image, as this is still an issue :)

Itxaka commented 1 year ago

Hello @liyimeng You are setting a random flavor name, so when framework tries to include the packages it tries searc hing for the given flavor in the profiles, but alpine is not a valid falvor. You should use either alpine-opensuse-leap or alpine-ubuntu as the ---FLAVOR to get an apline iso.

@Itxaka thanks! In this case, the we probably need to update document at kairos.io/docs/development/development, where it is misleading, what do u think?

Indeed this is an issue, we never updated the docs to show the current state of arm building.

Basically to build a generic arm iso (generic means its not tied to a device like a rpi4) you need to run a different target (earthly +all-arm-generic --FLAVOR=$FLAVOR) and also we dont mark that that has only been tested to work with opensuse-leap and under a VM in arm64 platforms, so not even on real hardware (is there any arm64 dev boxes out there?)

Will also update the docs to mention this. Thanks for the heads up!

liyimeng commented 1 year ago

it seem the build was not that reproducible. I try a couple of flavours. The one most close to succeed is alpine-ubuntu on amd64, it fails like this

          +framework | FLAVOR=alpine-ubuntu
          +framework | --> RUN luet cleanup --system-target /framework
          +framework |   ERROR    Error on removing 02
          +framework | ERROR Earthfile line 269:4
          +framework |       The command
          +framework |           RUN luet cleanup --system-target /framework
          +framework |       did not complete successfully. Exit code 2

================================== ❌ FAILURE ===================================

          +framework *failed* | Repeating the failure error...
          +framework *failed* | FLAVOR=alpine-ubuntu
          +framework *failed* | --> RUN luet cleanup --system-target /framework
          +framework *failed* |   ERROR    Error on removing 02
          +framework *failed* | ERROR Earthfile line 269:4
          +framework *failed* |       The command
          +framework *failed* |           RUN luet cleanup --system-target /framework
          +framework *failed* |       did not complete successfully. Exit code 2
🛰️ Reuse cache between CI runs with Earthly Satellites! 2-20X faster than without cache. Generous free tier https://cloud.earthly.dev

kairos-io / kairos

build fails on master and v2.4.1 on arm64 #1895