Multi-architecture support

[astefanutti]

At the moment, Camel K binaries and images are available for amd64 architecture only.

Following up on #1215, it would be useful to publish multi-architecture images, so that Camel K can be used on a wider range of architectures. Based on a recent PoC, I've identify the following tasks:

• Camel K:
  • [ ] Use a multi-architecture default Java base image
  • [ ] Build and publish multi-architecture images: BuildKit could be a good candidate to ease the task of building images for all target architectures, as it automatically picks the right base image and creates the multi-architecture manifest
  • [ ] The https://github.com/scylladb/go-set dependency doesn't compile on 32 bits architectures
• Kaniko:
  • [ ] Multi-architecture based image support: GoogleContainerTools/kaniko#646, GoogleContainerTools/kaniko#980
  • [ ] Multi-architecture warmer and executor images

[github-actions[bot]]

This issue has been automatically marked as stale due to 90 days of inactivity. It will be closed if no further activity occurs within 15 days. If you think that’s incorrect or the issue should never stale, please simply write any comment. Thanks for your contributions!

[robertonav20]

Any updates for this task? Can I help with something?

[squakez]

@robertonav20 you may have a look at the tasks in the issue description if you're willing to have a look at the implementation. A draft PR with some PoC may be useful to experiment and reason about the pros and cons.

[robertonav20]

Hi @squakez, I build and execute camel k operator and a camel quarkus application inside raspberry. I used docker buildx and specify the target platform.

I must modify makefile and dockerfile for camel-k-operator and use the correct java base image for camel quarkus application.

I apologize but I haven't strong skill with go language, I try to do my best. Can you give me an advice?

[squakez]

I think you can start having a look at the builder package which is the one in charge to take care for building images and pushing to registry.

[robertonav20]

Hi @squakez, i've tried to import BuildKit library like suggest @astefanutti inside the builder package but there is a problem of dependency

go: finding module for package github.com/go-logr/logr/funcr
github.com/apache/camel-k/pkg/builder imports
        github.com/moby/buildkit/client imports
        go.opentelemetry.io/otel/sdk/trace tested by
        go.opentelemetry.io/otel/sdk/trace.test imports
        github.com/go-logr/logr/funcr: module github.com/go-logr/logr@latest found (v1.2.3, replaced by github.com/go-logr/logr@v0.4.0), but does not contain package github.com/go-logr/logr/funcr

Now I'm trying to use buildah library instead buildKit, it's more friendly than buildkit, but it's the same I got error during build

# github.com/docker/docker/pkg/archive
/home/rob/go/pkg/mod/github.com/moby/moby@v0.7.3-0.20190826074503-38ab9da00309/pkg/archive/archive_unix.go:84:13: undefined: system.RunningInUserNS

you can find the my code here https://github.com/robertonav20/camel-k/... i will update you if i fix the problem

[robertonav20]

Hi, i have an update with this tasks, below the description for each point>

• Use a multi-architecture default Java base image

  I use the graalvm base image in order to use for amd64/arm64 like this FROM --platform=$BUILDPLATFORM ghcr.io/graalvm/graalvm-ce:ol8-java11-22.1.0

I thinked to use quarkus-mandrel-base-image for arm64 but isn't available for now (https://github.com/quarkusio/quarkus-images/commit/21e25a3c3ae3e3f4a4bd02b1e76f6051c31729c0)

• Build and publish multi-architecture images: BuildKit could be a good candidate to ease the task of building images for all target architectures, as it automatically picks the right base image and creates the multi-architecture manifest

  I modified the Makefile to use BuildKit with correct platform parameter in base of host's architecture

  docker buildx rm --all-inactive --force
  docker buildx create --append --name builder
  docker buildx build --platform=$(IMAGE_TARGET_PLATFORM) -t $(CUSTOM_IMAGE):$(CUSTOM_VERSION) -f build/Dockerfile .

So, for the first part can i submit pr?

For the integration-kit image, I cannot find a solution to build and push a working image because there are a problem dependency (read the previous comment). Do you have an idea how to fix?

[squakez]

Thanks for running those experiments and reporting the feedback. About the multi-architecture Java image, I guess we may wait for that to be officially available. We will align to Quarkus platform, so, that one will become eventually available. As for the operator container, I think we cannot proceed so easily to change the make images. We depend on that for the release process and likely the docker buildx is not available in the github actions we use for the official release.

However, you can create a new action beside images, ie, images-arch which can be run manually by anyone who wants to run that. We may eventually use that once we have the rest of pieces in place. @oscerd @astefanutti @phantomjinx wdyt?

[oscerd]

It makes sense to me. +1.

[robertonav20]

So, @squakez suggest a command makes-arch to build for other platform, we can make the same thing with dockerfile because the base image isn't available. It makes sense if that image doesn't available soon, are you right? I don't know if can be a good idea, maybe there are other constraints for quarkus.

I checked the GitHub action, here https://github.com/marketplace/actions/build-and-push-docker-images can be found the official docker GitHub action of docker which support buildx, so I think it's possible use it.

Anyway, It's a pleasure help the community.... For now I try to build an integration kit multi arch with buildah strategy.

It's possible just update buildah version to 1.23.3 because the latest version support arm64 arch... Stay tuned I will give some feedback soon

[robertonav20]

As i say previously, i build and execute an integration kit inside rpi4 8gb with architecture arm64 with the use of buildah, below the modifies:

• Update buildah at 1.23.3 version
• Add the parameter platform

  bud := []string{
      "buildah",
      "bud",
      "--storage-driver=vfs",
      "--platform",
      platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant,
      "--pull-always",
      "-f",
      "Dockerfile",
      "-t",
      task.Image,
      ".",
  }

All image are available here https://hub.docker.com/repository/docker/robnav24241/camel-k operator and integration-kit

[squakez]

Nice work. Ideally those architecture values should be "parameter-ized" so that the user can provide that information somewhere (ie, in the IntegrationPlatform configuration).

[robertonav20]

@squakez sorry, I don't understand why should be parameterized? The integration kit are builded inside kubernetes so with architecture of the node amd64, arm64 etc.... this thing can't be modified by the user.

Can you describe me?

[squakez]

Yeah, I am thinking on a generic situation where you have multiple nodes in your cluster, and the operator is running on any non-ARM architecture. Then you have a special node in ARM architecture where you are willing to run the Integrations. In such case, having that value as a parameter would help.

[robertonav20]

Ok Great! Thanks! But you are limited by buildah architecture support, for others solutions it's necessary fix the dependency problem which i describe some comment ago

Anyway, i saw the IntegrationPlatform struct is based entirely by external attributes fromapimachinery@v0.22.5, so i suggest add the Architeture field inside builderTrait and PublishTask like this

// The builder trait is internally used to determine the best strategy to
// build and configure IntegrationKits.
//
// +camel-k:trait=builder.
type builderTrait struct {
    BaseTrait `property:",squash"`
    // the architecture of image
    Architecture string `json:"baseImage,omitempty"`
    // Enable verbose logging on build components that support it (e.g. Kaniko build pod).
    Verbose *bool `property:"verbose" json:"verbose,omitempty"`
    // A list of properties to be provided to the build task
    Properties []string `property:"properties" json:"properties,omitempty"`
}

// PublishTask image publish configuration
type PublishTask struct {
    // can be useful to share info with other tasks
    ContextDir string `json:"contextDir,omitempty"`
    // Architecture of image
    Architecture string `json:"baseImage,omitempty"`
    // base image layer
    BaseImage string `json:"baseImage,omitempty"`
    // final image name
    Image string `json:"image,omitempty"`
    // where to publish the final image
    Registry RegistrySpec `json:"registry,omitempty"`
}

Then can be map Architecture when necessary just like this

    case v1.IntegrationPlatformBuildPublishStrategyBuildah:
        if t.Architecture == "" {
            t.Architecture = platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant
        }

        e.BuildTasks = append(e.BuildTasks, v1.Task{Buildah: &v1.BuildahTask{
            BaseTask: v1.BaseTask{
                Name: "buildah",
            },
            PublishTask: v1.PublishTask{
                Architecture: t.Architecture,
                Image:        getImageName(e),
                Registry:     e.Platform.Status.Build.Registry,
            },
            Verbose: t.Verbose,
        }})

PublishTask struct is shared by all builders for this reason i choosed her

@squakez @oscerd @astefanutti @phantomjinx do you agree?

[squakez]

You can have a look at some refactoring we've done recently here https://github.com/apache/camel-k/pull/3032 - We've introduced a generic PublishStrategyOptions in the IntegrationPlatform that can be used for any strategy extra option (we're using for Kaniko there). I think we could leverage that and from that onward cascade the parameter up to the Buildah Task. Ideally we don't want a specific parameter to slip into an abstract concept (such as the Builder). Also, feel free to create any draft PR and we can discuss over there any idea you have around this or other developments.

[robertonav20]

Ok @squakez, i understand your advice but i dont know how to use PublishStrategyOptions.

Can you give me and example to fill that structure from cmd?

[squakez]

Sure, you can have a look at #3032 where we used that for the "Kaniko" options. We may mimick that and include any "Buildah" option as well.

[robertonav20]

Hi @squakez, yes i have a look yesterday, below i describe the code:

Inside the package builder i add buildah.go file to manage buildah constants properties

package builder

const BuildahPlatform = "BuildahPlatform"

Here you can find the mapping to buildah builder

    case v1.IntegrationPlatformBuildPublishStrategyBuildah:
        var architecture string
        var found bool
        if architecture, found = e.Platform.Status.Build.PublishStrategyOptions[builder.BuildahPlatform]; !found {
            architecture = platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant
        }

        e.BuildTasks = append(e.BuildTasks, v1.Task{Buildah: &v1.BuildahTask{
            BaseTask: v1.BaseTask{
                Name: "buildah",
            },
            PublishTask: v1.PublishTask{
                Architecture: architecture,
                Image:        getImageName(e),
                Registry:     e.Platform.Status.Build.Registry,
            },
            Verbose: t.Verbose,
        }})

Then map architecture attribute to buildah command.

I see a difference between https://github.com/apache/camel-k/pull/3032 and this task, that task introduce a command parameter for install, but in our case we must manage the architecture of build for integration kit, so it's necessary introduce this parameter inside run command

My questions are:

• Where is the point of the code ? I suppose run.go inside pkg/cmd and here i can add the new buildah parameter (exmaple: buildah-platform)
• The variable platform inside run.go method createOrUpdateIntegration is empty, How to fill it ?
• How to check if that map PublishStrategyOptions contains that parameter?

I'm hope you understand my questions.

Thanks in advance

[squakez]

Any build configuration should be included into the IntegrationPlatform, not exposed in the kamel run command. I think you can extend the BuildahTask to include the new needed parameters https://github.com/apache/camel-k/blob/fe997b9ac1297884840ffb42bb41302bb5040ee5/pkg/apis/camel/v1/build_types.go#L107 and passing them from IntegrationPlatform.PublishStrategyOptions as we're doing for Kaniko settings. Once they are in the BuildahTask, you should be able to use them.

[robertonav20]

Ok thanks, i understand your advice and i fix the code just like this.

type BuildahTask struct {
    BaseTask    `json:",inline"`
    PublishTask `json:",inline"`
    // The platform of build image
    Platform string `json:"baseImage,omitempty"`
    // log more information
    Verbose *bool `json:"verbose,omitempty"`
}

case v1.IntegrationPlatformBuildPublishStrategyBuildah:
    var platform string
    var found bool
    if platform, found = e.Platform.Status.Build.PublishStrategyOptions[builder.BuildahPlatform]; !found {
        platform = platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant
    }

    e.BuildTasks = append(e.BuildTasks, v1.Task{Buildah: &v1.BuildahTask{
        Platform: platform,
        BaseTask: v1.BaseTask{
            Name: "buildah",
        },
        PublishTask: v1.PublishTask{
            Image:    getImageName(e),
            Registry: e.Platform.Status.Build.Registry,
        },
        Verbose: t.Verbose,
    }})

In this way you can manage the platform parameter if you want, just fill PublishStrategyOptions with the correct paramater from outside.

But, i don't understand how to works PublishStrategyOptions, for kaniko i found this code inside install.go

kanikoBuildCacheFlag := cobraCmd.Flags().Lookup("kaniko-build-cache")
if kanikoBuildCacheFlag.Changed {
    platform.Spec.Build.PublishStrategyOptions[builder.KanikoBuildCacheEnabled] = strconv.FormatBool(o.KanikoBuildCache)
}

If i undertstand well, with install command you can pass kaniko-build-cache parameter and to set the PublishStrategyOptions, right? There is another way to fill PublishStrategyOptions? How do you thing to manage the platform parameter for buildah?

Thanks in advance! 😄

[squakez]

Nice, that fits perfectly into the design. I think those parameters must be provided by the user editing the IntegrationPlatform (manually) ie kubectl edit ip. A nice addition would be to transform that kaniko-build-cache into something generic, so we can manage any PublishStrategyOptions option. I'm creating a follow up issue for that. For now, I guess that the manual editing of the IntegrationPlatform would be enough for the scope.

[robertonav20]

Ok great! So you leave to the user edit the manifest of integration platform under Camel K CRD.

Can i submit a PR?

[squakez]

Sure, it's more than welcome.

[robertonav20]

You can find here https://github.com/apache/camel-k/pull/3309 the pr

[barius]

I'm able to build a arm64 camel-k-operator image by first creating a arm64 quarkus-mandel image, then base camel-k on this. However, integration kits built in this arm64 operator is still amd64/x64. I'm running Docker Desktop on an M1 Macbook, camel-k 1.9.1.

Looked into camel-k builds and saw the Spectrum builder. Don't know if it supports arm64. Is there any way I can get a arm64 image? Maybe after the above PR is merged? Or is there anyway I can help with that?

[robertonav20]

Hi @barius to build integration kits arm64 it's necessary use buildah as build-strategy you can follow the pr open from this ticket

[barius]

@robertonav20 Okay I'll try it! Good work!

[barius]

Confirm that the PR works! Now my integrations are arm64, yeah! Saw ~4x performance on my mac :) I'll still using my local arm64 quarkus-mandrel image though, didn't find the manual command to build kamel operator to support arm architecture :(

[robertonav20]

Ok great you can find the full guide here https://github.com/robertonav20/camel-k/blob/main/docs/modules/ROOT/pages/configuration/multi-architecture.adoc

[barius]

Ah I see, great! However it uses Graal. Don't know what the difference between mandrel and graal so I'm waiting for the official release :) For now everything works for me and I don't have much time testing it out :P

[squakez]

Partially available in #3309

[robertonav20]

Can we close this issue?

[squakez]

No, I think that although partially available with your development, this issue need some more work to be fully completed.

[barius]

Although Buildah is perfectly fine with regards to building kit images, when Istio auto-injection is enabled, the camel-k builder always wait for the Istio container to complete (which will never happen), and triggers the default 5 min timeout (though Buildah finished in around 1~2 min and after the timeout the build still succeeds).

Don't know if this is the expected behavior, or if I should open another issue with it.

[squakez]

@barius please better open a new issue and which version is affected (I suppose the nightly build).

[barius]

@squakez yes pls take a look #3408

[robertonav20]

@squakez now the is available the multi image of ubi-quarkus-mandrel and ubi-quarkus-micro-image, so you can update the dockerfile to support arm64 natively then delete the new file Dockerfile.arch.

Here the quay.io repository podman pull quay.io/quarkus/quarkus-micro-image:2.0 podman pull quay.io/quarkus/ubi-quarkus-mandrel:22.0.0.2-Final-java11-arm64

Ref. to the closed issue https://github.com/quarkusio/quarkus-images/issues/83

[squakez]

Cool! Thanks for the heads up. Feel free to have a look yourself while this is not picked up for development.

[tobiasoort]

Edit: It seems I was being captain obvious. Is there anything I can help with?

For future googlers, the following error in your pod's logs: standard_init_linux.go:228: exec user process caused: exec format error means you're running the wrong arch (in this specific case, running an amd64 image on arm64).

[squakez]

Edit: It seems I was being captain obvious. Is there anything I can help with?

For future googlers, the following error in your pod's logs: standard_init_linux.go:228: exec user process caused: exec format error means you're running the wrong arch (in this specific case, running an amd64 image on arm64).

I guess the last message from @robertonav20 could be a good point from where to continue with the development of this issue.

[tobiasoort]

So, based on the assumptions:

• ubi-quarkus-mandrel v22.0.0.2-Final-java11 is the version to target (v21 doesnt have arm builds) ✅
• hosting the application in the new quarkus container is enough - the Maven build doesnt need to change (yay jvm?) ✅
• I know how buildx works ✅

I have submitted https://github.com/apache/camel-k/pull/3820

[tobiasoort]

• I know how buildx works

Foreshadowing quote was exactly that. I'm afraid i'll break the build because with multi-arch images we can't rely on the local image store (see https://github.com/docker/buildx/issues/59). I've worked around it and hope that none of the processes rely on 'the image just being there'.

Also, I see a 'docker build' step for a 'bundle'. What is that 'bundle', and do I need to fix that too?

[tobiasoort]

Edit: this was wrong.

[tobiasoort]

So, the issue i'm currently blocked by is:

• The IMAGE_NAME is set in the secrets.E2E_CLUSTER_CONFIG secret to kind-registry:5000/apache/camel-k (I think, its a secret!)
• The IMAGE_NAME contains the hostname kind-registry or that name comes from somewhere
• This hostname is not resolving at all (it is supposed to be 127.0.0.1) and thus it errors out.

How can we validate this? Can someone share the outline of the secrets.E2E_CLUSTER_CONFIG so I can override it in my own fork? Edit: it was networking related. Buildx lives inside of a container and to resolve the kind-registry we need hostnetworking. Added a small change, lets see if the pipeline-deities are now happy.

[tobiasoort]

Integration tests are failing due to:

 Error: p.go:274:     > {"level":"error","ts":1668698213.7254713,"logger":"camel-k.maven.build","msg":"\t[error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.IllegalStateException: Out of date version of GraalVM detected: native-image 22.0.0.2-Final Mandrel Distribution (Java Version 11.0.14+9). Quarkus currently supports 22.3.0. Please upgrade GraalVM to this version."

I don't really understand this - I cant find a version reference of GraalVM in the source.

[squakez]

g":"\t[error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.IllegalStateException: Out of date version of GraalVM detected: native-image 22.0.0.2-Final Mandrel Distribution (Java Version 11.0.14+9). Quarkus currently supports 22.3.0. Please upgrade GraalVM to this version."

Known issue #3817

[tobiasoort]

Current status: ready for polishing. All pipeline errors are due to #3817 or timeouts that are also already known. Next up: parameterize the build archs.

[tobiasoort]

Current status: retracted the PR because of the amount of noise. The main branch is broken at the moment so i'll wait for things to green up again and then i'll continue getting this thing done.

[squakez]

As we're planning to rework the way we build the images, I think we should take the opportunity to include the multi-architecture support as well.