Unable to build / deploy on ARM

zwan2 commented 2 years ago

 - deployment/adservice: container server terminated with exit code 1
    - pod/adservice-5b65c5ff6-zzfkl: container server terminated with exit code 1
      > [adservice-5b65c5ff6-zzfkl server] Error occurred during initialization of VM
      > [adservice-5b65c5ff6-zzfkl server] Could not find agent library /opt/cprof/profiler_java_agent.so in absolute path, with error: /opt/cprof/profiler_java_agent.so: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)
 - deployment/adservice failed. Error: container server terminated with exit code 1.

xtineskim commented 2 years ago

Hello! Thanks for bringing this up!! At first glance, it seems like it might be a minikube issue, could you specify your environment (os, minikube version, anything else that may be relevant)? Thanks!

zwan2 commented 2 years ago

m1 mac Big Sur 11.6x
minikube:latest using docker

It may be resolved when x86 condition

jxlwqq commented 2 years ago

Does not support deployment it under arm64 architecture, such as Apple chip's Mac, Raspberry Pi. @zwan2

Are there any plans to support arm64? @ckim328

NimJay commented 2 years ago

tl;dr - We will not be (consciously) supporting the deployment of our microservice sample images on ARM anytime soon. I say "consciously" because you might still be able to build the Docker images locally (and target ARM) and get them to deploy on ARM machines.

Hi @zwan2 and @jxlwqq! Thank you for raising this issue! 😁

We've actually been talking about ARM support a lot recently. This issue is becoming a growing concern — especially given the popularity of the M1 chip.

Currently, the Docker images (for each microservice) that we host (e.g., gcr.io/google-samples/microservices-demo/checkoutservice:v0.3.1) use AMD64 architecture.

> docker image inspect gcr.io/google-samples/microservices-demo/checkoutservice:v0.3.1 | grep Arch
"Architecture": "amd64",

And, unfortunately, we do not intend to create sample images that use ARM architecture anytime soon — mainly because Google Kubernetes Engine does not currently support ARM-based image deployments.

@zwan2 In your Kubernetes deployments, are you using the images hosted at gcr.io/google-samples/microservices-demo/...? If so, have you tried building the images locally (and targeting ARM) and using those locally-built Docker images in minikube? I can't guarantee that this works (because I am not sure if the base images used in each of microservices' Dockerfiles support ARM). But it's worth trying. :)

bourgeoisor commented 2 years ago

Not fully supported yet, but note that you can alternatively use docker buildx from your ARM machine to build amd64 images: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images

We're currently investigating the use of ARM chips by developers to build our sample applications.

xtineskim commented 2 years ago

cooling down out of our oslo, still investigations being done for the use of ARM

NimJay commented 2 years ago

@zwan2, @jxlwqq

GKE (Google Kubernetes Engine) now supports the ability to run containerized workloads using the Arm architecture. :) These resources might be of interest to you:

I have not tested your specific user journey myself (i.e., I have not run Arm-based container images on GKE, built on Mac's M1 chips). But I hope this info helps.

minherz commented 2 years ago

@zwan2 , @jxlwqq would you mind to try the tutorial for deploying Online Boutique sample application described in the official documentation? Please, let us know if it does not work. Deploying the application does not require building containers.

smazzone commented 2 years ago

just checking if there are plans to build and publish arm64 images in the default repo for people who are willing to run the demo locally (i.e. M1 and docker desktop or similar). Thanks again.

mathieu-benoit commented 2 years ago

One more evidence we need to support this: https://github.com/GoogleCloudPlatform/microservices-demo/issues/810

mathieu-benoit commented 2 years ago

Another evidence here: https://github.com/GoogleCloudPlatform/microservices-demo/issues/832

chenhy97 commented 2 years ago

need the ARM architecture supports!

GregCKrause commented 1 year ago

Not fully supported yet, but note that you can alternatively use docker buildx from your ARM machine to build amd64 images: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images

We're currently investigating the use of ARM chips by developers to build our sample applications.

Following up on @bourgeoisor's response.

To use this with skaffold, set build.local.useBuildkit to true in skaffold.yaml:

build:
  ...
  local:
    useBuildkit: true

smazzone commented 1 year ago

@GregCKrause I have been running the demo on arm64 for a couple of weeks. I have also used buildx to build images on both architectures. But, changing skaffold.yaml was not enough. I had to slightly re-adapt Dockerfile as health probes are hardcoded for amd64 architecture see example here

RUN GRPC_HEALTH_PROBE_VERSION=v0.4.14 && \ wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 && \ chmod +x /bin/grpc_health_probe

I hope this helps. Stefano

minherz commented 1 year ago

I will keep this issue open for now and mark it as a "good first issue" for someone who wants to submit a kustomize or Helm configuration that will build and deploy on ARM platform.

NimJay commented 1 year ago

Update (Jan 2023)

Supporting ARM is definitely a priority for 2023.

More specifically:

A user should be able to run Online Boutique in development mode (skaffold dev), on an ARM-based device.
A user should be able to deploy Online Boutique, as is, on an ARM-based Kubernetes cluster. In this sentence, “as is” means that the user does not have to build Online Boutique’s container images themselves. Official ARM-supporting container images should be published by Online Boutique.

We hope to achieve this by the end of this year (2023).

Thank you, @smazzone and @GregCKrause for the tips you've shared — those will be helpful.

odidev commented 1 year ago

I have made changes in the Dockerfiles to build the images for arm64 platform and now skaffold is running successfully and pods are in running state.

I have used my docker images that I built for arm64 in kubernetes-manifests.yaml and deployed on minikube cluster. Now all the pods are running successfully but when applying the same kubernetes-mainfests.yaml file in GKE cluster the pods are in pending state for arm64.

I have also built the images using buildx and the pods are running for amd64 on GKE cluster but for arm64 it is still in pending state. Can you please share your feedback regarding the same?

NimJay commented 1 year ago

Heads-up:

A comment above mentioned:

I had to slightly re-adapt Dockerfile as health probes are hardcoded for amd64 architecture see example here

This additional step (of swapping the grpc-health-probe binaries deployed next to Online Boutique's gRPC microservices) is no longer necessary, since the merge of https://github.com/GoogleCloudPlatform/microservices-demo/pull/1837. In other word, the grpc-health-probe binary is no longer used.

mtardy commented 5 months ago

Update (Jan 2023)

Supporting ARM is definitely a priority for 2023. [...] We hope to achieve this by the end of this year (2023).

Hey, we bumped into this, wanting to use the demo as part of our testing pipeline that runs on arm64 https://github.com/cilium/tetragon/pull/2345. Do you still want to publish arm64 images anytime soon? Thanks!

bourgeoisor commented 5 months ago

Hi @mtardy! It's something we wanted to do last year but prioritization went a different direction.

cc: @NimJay might be able to give some more details / if there's a plan for this year.

But yeah this is still something we'd like to do.

willfindlay commented 5 months ago

Are you looking for any help on this or is it more of a case of "we could do this but aren't sure we're in place to maintain another set of images right now"?

maitrungduc1410 commented 3 months ago

building cartservice on Apple M2 hang at restore

Dockerfile

FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/sdk:8.0.302-noble@sha256:bd836d1c4a19860ee61d1202b82561f0c750edb7a635443cb001042b71d79569 as builder
WORKDIR /app
COPY cartservice.csproj .
RUN dotnet restore cartservice.csproj \
    -r linux-x64
COPY . .
RUN dotnet publish cartservice.csproj \
    -p:PublishSingleFile=true \
    -r linux-x64 \
    --self-contained true \
    -p:PublishTrimmed=true \
    -p:TrimMode=full \
    -c release \
    -o /cartservice

# https://mcr.microsoft.com/product/dotnet/runtime-deps
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/runtime-deps:8.0.6-noble-chiseled@sha256:55d6e41f2e7687c597daa4fdca997b07beb3e23b6283729e19bb8ceb272def1a

WORKDIR /app
COPY --from=builder /cartservice .
EXPOSE 7070
ENV DOTNET_EnableDiagnostics=0 \
    ASPNETCORE_HTTP_PORTS=7070
USER 1000
ENTRYPOINT ["/app/cartservice"]

Logs:

[+] Building 552.2s (10/14)                    docker:desktop-linux
 => [internal] load build definition from Dockerfile           0.0s
 => => transferring dockerfile: 1.51kB                         0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/run  0.1s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk  0.1s
 => [internal] load .dockerignore                              0.0s
 => => transferring context: 92B                               0.0s
 => [builder 1/6] FROM mcr.microsoft.com/dotnet/sdk:8.0.302-n  0.0s
 => [stage-1 1/3] FROM mcr.microsoft.com/dotnet/runtime-deps:  0.0s
 => [internal] load build context                              0.0s
 => => transferring context: 569B                              0.0s
 => CACHED [stage-1 2/3] WORKDIR /app                          0.0s
 => CACHED [builder 2/6] WORKDIR /app                          0.0s
 => CACHED [builder 3/6] COPY cartservice.csproj .             0.0s
 => [builder 4/6] RUN dotnet restore cartservice.csproj      552.0s

I'm wondering, if this issue can't be resolved (it's there for years), why not convert this cartservice (and other services which have issue with Apple chip) to other languages that works for everyone? 🤔🤔🤔

maitrungduc1410 commented 3 months ago

finally I got it working, we need to set ENV DOTNET_EnableWriteXorExecute=0

My Dockerfile

FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/sdk:8.0 as builder
ENV DOTNET_EnableWriteXorExecute=0
WORKDIR /app
COPY cartservice.csproj .
RUN dotnet restore cartservice.csproj \
    -r linux-x64
COPY . .
RUN dotnet publish cartservice.csproj \
    -p:PublishSingleFile=true \
    -r linux-x64 \
    --self-contained true \
    -p:PublishTrimmed=true \
    -p:TrimMode=full \
    -c release \
    -o /cartservice

# https://mcr.microsoft.com/product/dotnet/runtime-deps
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/runtime-deps:8.0

WORKDIR /app
COPY --from=builder /cartservice .
EXPOSE 7070
ENV DOTNET_EnableDiagnostics=0 \
    ASPNETCORE_HTTP_PORTS=7070
USER 1000
ENTRYPOINT ["/app/cartservice"]

Update: after the fix I still sometime face this issue:

20240706-123737

I ended up fixing the issue by using alpine image:

FROM mcr.microsoft.com/dotnet/sdk:8.0 as builder
WORKDIR /app
COPY cartservice.csproj .
RUN dotnet restore cartservice.csproj -r linux-musl-arm64
COPY . .
RUN dotnet publish cartservice.csproj -p:PublishSingleFile=true -r linux-musl-arm64 --self-contained true -p:PublishTrimmed=True -p:TrimMode=Link -c release -o /cartservice --no-restore

# https://mcr.microsoft.com/v2/dotnet/runtime-deps/tags/list
FROM mcr.microsoft.com/dotnet/runtime-deps:8.0-alpine3.18-arm64v8

WORKDIR /app
COPY --from=builder /cartservice .
EXPOSE 7070
ENV DOTNET_EnableDiagnostics=0 \
    ASPNETCORE_URLS=http://*:7070
USER 1000
ENTRYPOINT ["/app/cartservice"]

It comes from this PR: https://github.com/GoogleCloudPlatform/microservices-demo/pull/2266/files#diff-c9094dcf2dd2b5e3bb57d27631699d50e63b9bfb3aa3665db826e031b0ab9b4c

mtardy commented 3 months ago

Update (Jan 2023)

Supporting ARM is definitely a priority for 2023. [...] We hope to achieve this by the end of this year (2023).

Hey, we bumped into this, wanting to use the demo as part of our testing pipeline that runs on arm64 cilium/tetragon#2345. Do you still want to publish arm64 images anytime soon? Thanks!

By the way, I hope it's not inappropriate to write about "alternatives", but for people looking for similar demos working both on amd64 and arm64, we ended up using https://github.com/open-telemetry/opentelemetry-demo.

GoogleCloudPlatform / microservices-demo