search

NetApp / trident

Storage orchestrator for containers
Apache License 2.0
759 stars 220 forks source link

Add non-amd64 and install details to BUILD.md #534

Open scaleoutsean opened 3 years ago

scaleoutsean commented 3 years ago

Describe the solution you'd like

The current build instructions seem to work for amd64 only.

It'd be nice to have BUILD.md document steps to build and deploy for another architecture (no changes in code would be necessary), so that those interested in experimenting on other platforms can more easily give it a try.

Of course even better would be to modify the Makefiles and Dockerfiles to make that possible, but that involves comparatively more work and official support for those architectures so I'm instead asking for a basic Doco improvement and not submitting a build-related Bug.

I think the below build procedure works (I haven't tried to install yet), but I don't know if it's strictly necessary or maybe there's a simpler way to get this done. Also, if after tridentctl, trident_operator and container are built, any steps are different from manual deployment x86_64, that could be added as well.

Additional context

Following the process in BUILD.md, I get a30130f9efd9 created. But no matter what I try, I couldn't get it deployed. I tried native build, cross-platform build, Docker and go-based builds, different versions, nothing worked. The problem was that no matter how I build, the container image is always tagged in the amd64 format.

Finally I started looking at Dockerfiles.

$ docker images
REPOSITORY                           TAG                                        IMAGE ID            CREATED             SIZE
trident                              21.01.0-arm64                              a30130f9efd9        19 minutes ago      85.1MB
golang                               1.14                                       c7909254f92e        9 days ago          715MB
alpine/helm                          latest                                     584a8902bb56        13 days ago         46MB
quay.io/coreos/flannel               v0.13.1-rc2                                25dbdd806019        2 weeks ago         63.6MB
k8s.gcr.io/kube-proxy                v1.20.2                                    788e63d07298        5 weeks ago         116MB
k8s.gcr.io/kube-controller-manager   v1.20.2                                    3a1a2b528610        5 weeks ago         107MB
k8s.gcr.io/kube-apiserver            v1.20.2                                    95d99817fc33        5 weeks ago         113MB
k8s.gcr.io/kube-scheduler            v1.20.2                                    60d957e44ec8        5 weeks ago         43.3MB
registry                             2                                          357aab9e21a8        2 months ago        24.5MB
k8s.gcr.io/etcd                      3.4.13-0                                   05b738aa1bc6        5 months ago        312MB
k8s.gcr.io/coredns                   1.7.0                                      db91994f4ee8        8 months ago        42.8MB
k8s.gcr.io/pause                     3.2                                        2a060e2e7101        12 months ago       484kB
gcr.io/distroless/static             480a6179aef97ba11817a86d1c7a0fed1e07e4b3   7b715d870c4e        51 years ago        1.79MB

# My trident container image

$ docker inspect a30130f9efd9 | grep Arch
        "Architecture": "amd64",

# Revisit one of few remaining suspects - gcr.io/distroless/static, base image in Trident Dockerfiles.
$ docker inspect 7b715d870c4e | grep Arch
        "Architecture": "amd64",

# Hmmm, let's check its issues and bingo:
# https://github.com/GoogleContainerTools/distroless/issues/583

# switch to an ARM64 image
$ docker pull gcr.io/distroless/static@sha256:8ea529b8f58b0a8f8c9197ceec0188a34c00861d3a88d5b0e51bcfc4a7bd2137

# docker inspect f0a9eb72f317 | grep Arch
        "Architecture": "arm64",

# modify all 3 Dockerfiles in repo to use it
# try again

$ GOOS=linux GOARCH=arm64 make trident_build

# docker images
REPOSITORY                           TAG                                        IMAGE ID            CREATED             SIZE
trident                              21.01.0-arm64                              08310958adfd        8 seconds ago       85.1MB
golang                               1.14                                       c7909254f92e        9 days ago          715MB
alpine/helm                          latest                                     584a8902bb56        13 days ago         46MB
quay.io/coreos/flannel               v0.13.1-rc2                                25dbdd806019        2 weeks ago         63.6MB
k8s.gcr.io/kube-proxy                v1.20.2                                    788e63d07298        5 weeks ago         116MB
k8s.gcr.io/kube-apiserver            v1.20.2                                    95d99817fc33        5 weeks ago         113MB
k8s.gcr.io/kube-controller-manager   v1.20.2                                    3a1a2b528610        5 weeks ago         107MB
k8s.gcr.io/kube-scheduler            v1.20.2                                    60d957e44ec8        5 weeks ago         43.3MB
registry                             2                                          357aab9e21a8        2 months ago        24.5MB
k8s.gcr.io/etcd                      3.4.13-0                                   05b738aa1bc6        5 months ago        312MB
k8s.gcr.io/coredns                   1.7.0                                      db91994f4ee8        8 months ago        42.8MB
k8s.gcr.io/pause                     3.2                                        2a060e2e7101        12 months ago       484kB
gcr.io/distroless/static             <none>                                     f0a9eb72f317        51 years ago        1.79MB
gcr.io/distroless/static             480a6179aef97ba11817a86d1c7a0fed1e07e4b3   7b715d870c4e        51 years ago        1.79MB

# check Trident build from latest build atempt
$ docker inspect 08310958adfd | grep Arch
        "Architecture": "arm64",

Anyone wanting to build for a non-amd64 arch could just get the right gcr.io/distroless/static image for their Dockerfile.

scaleoutsean commented 3 years ago

I completed the rest, basically there's just other 2 tricky places:

More here.

gnarl commented 3 years ago

@scaleoutsean, ARM64 support is on the Trident roadmap and will be supported in a future release.

skuethe commented 2 years ago

Any update on this? Looking forward for multi architecture builds :)

scaleoutsean commented 2 years ago

@skuethe you're likely looking for an officially supported version, but if you want to experiment with current version, give it a try... I've been running various releases (from v21.01 to current) on different versions of vanilla K8s for over a year now and didn't notice anything out of ordinary. I use iSCSI in my environment.

skuethe commented 2 years ago

thank you for your feedback! We are currently in the process of testing on s390x arch :)

scaleoutsean commented 2 years ago

@christianjedroCDT, I somehow got your comment asking about additional details via email (Github email notifications), but I can't see it here. Yes, I documented the details (above) but even even more detail on my blog, but I didn't want to "advertise" it. Now that you ask, for ARM64 the above is likely enough to get everything working. For (Linux on) PowerPC and S390 I had to make a small patch.

christianjedroCDT commented 2 years ago

@scaleoutsean actually i´ve asked if you have detailed info on how to do it but then found that you´ve already provided the link to your article, which i´ve anyway found before through google.. then deleted the comment once realizing it.

One note to your article - i´ve actually deployed an image that was built with your guide to s390x and it seems to be working fine. Actually waiting to be able to create a backend but it looks promising.