feat(riscv64): Here is how far I've got.

[IngwiePhoenix]

Hello!

I have been using k3s for a while now, but long-term I want to expand to literally all my hardware - which includes a VisionFive2. So, I just throw stuff at a wall and see what sticks. Hence, here is an attempt at compiling for riscv64:

root@riscboi /n/o/k0s (main)# make HOST_ARCH=riscv64 EMBEDDED_BINS_BUILDMODE=none -j$(nproc)
mkdir -p -- 'build/cache'
docker build --progress=plain --iidfile '.k0sbuild.docker-image.k0s' \
  --build-arg BUILDIMAGE=docker.io/library/golang:1.22.4-alpine3.20 \
  -t k0sbuild.docker-image.k0s - <build/Dockerfile
STEP 1/3: FROM docker.io/library/golang:1.22.4-alpine3.20
Trying to pull docker.io/library/golang:1.22.4-alpine3.20...
Getting image source signatures
Copying blob 4f4fb700ef54 done   |
Copying blob 4e5b5f435f7e done   |
Copying blob d4714cc4c8bb done   |
Copying blob 720f2671938f done   |
Copying blob e88f851c95d1 done   |
Copying config f9e44e4509 done   |
Writing manifest to image destination
STEP 2/3: RUN apk add --no-cache make gcc musl-dev binutils-gold
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/main/riscv64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/community/riscv64/APKINDEX.tar.gz
ERROR: unable to select packages:
  binutils-gold (no such package):
    required by: world[binutils-gold]
Error: building at STEP "RUN apk add --no-cache make gcc musl-dev binutils-gold": while running runtime: exit status 1
make: *** [Makefile:108: .k0sbuild.docker-image.k0s] Error 1

Notes:

• This is a Debian rolling version (calls itself "trixie/sid").
• This isn't Docker, it's Podman. I just have the Docker wrappers.
• CRI is runc

I thought setting EMBEDDED_BINS_BUILDMODE=none would disable Docker alltogether, but it did not...?

Any improvements I could do to get this building a little further than...that?

Kind regards, Ingwie

[twz123]

/cc #1919

haven't tested it in a while, but now that Alpine 3.20 has RISC-V support, you should be able to compile k0s directly on the VisionFive using Docker. It'll take time, but it should work.

Cross compilation is not really supported by the k0s build system. Sorry. You can disable the docker wrapper by make GO=go K0S_EMBEDDED_BINS=none. Note that you'll need RISC-V docker images as well if you want to run workloads on the cluster.

[twz123]

That was my last attempt: https://github.com/k0sproject/k0s/issues/1919#issuecomment-1746962060

You can have a look at the diff to figure out what to tweak.

[twz123]

@ncopa concerning https://github.com/k0sproject/k0s/issues/1919#issuecomment-1761203870, do you think we can simply unconditionally switch from binutils-gold to binutils?

[ncopa]

Go forces use of gold on arm/arm64 due to an old bug in binutils, which seems to be fixed now. Go still need to adapt.

See: https://github.com/golang/go/issues/22040

I think we can pull in binutils-gold conditionally.

[github-actions[bot]]

The issue is marked as stale since no activity has been recorded in 30 days

[IngwiePhoenix]

Hello!

Apologies for the very late response. I got swamped with work and after I moved and reinstalled my RISC-V board - VisionFive2 - I gave it another shot since I saw the ongoing messages here. It actually got shockingly close to finish building!

+++ [0823 06:41:17] Placing binaries
+ mv '/go/src/github.com/kubernetes/kubernetes/_output/local/bin/*/*/kubelet' /out/
mv: cannot stat '/go/src/github.com/kubernetes/kubernetes/_output/local/bin/*/*/kubelet': No such file or directory
Error: building at STEP "RUN set -ex;   export GOPATH=/go;   if [ "${TARGET_OS}" = windows ]; then     commands="${KUBERNETES_BINS}";     binarySuffix=.exe;     export KUBE_BUILD_PLATFORMS=windows/amd64;   else     commands="${KUBERNETES_BINS}";     binarySuffix='';   fi;   if [ ${BUILD_GO_CGO_ENABLED:-0} -eq 1 ]; then     export KUBE_CGO_OVERRIDES=$commands;   else     export KUBE_STATIC_OVERRIDES=$commands;   fi;   mkdir /out;   export SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH;   export FORCE_HOST_GO=y;   export KUBE_VERBOSE=9;   export KUBE_GIT_VERSION="v$VERSION+k0s";   for cmd in $commands; do     make GOFLAGS="${BUILD_GO_FLAGS} -tags=${BUILD_GO_TAGS}" GOLDFLAGS="${BUILD_GO_LDFLAGS_EXTRA}" WHAT=cmd/$cmd;     mv /go/src/github.com/kubernetes/kubernetes/_output/local/bin/*/*/$cmd$binarySuffix /out/;   done": while running runtime: exit status 1
make[1]: *** [Makefile:103: .docker-image.kubernetes.stamp] Error 1
make[1]: Leaving directory '/nvme/opt/k0s/embedded-bins'
make: *** [Makefile:209: .bins.linux.stamp] Error 2

Unfortunately I don't see an easy way of checking that file as the build command does not mound that particular folder anywhere:

docker build --progress=plain --iidfile '.docker-image.kubernetes.stamp' -t k0sbuild.docker-image.kubernetes:latest --build-arg TARGET_OS=linux --build-arg CONTAINERD_BINS="containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2" --build-arg KUBERNETES_BINS="kubelet kube-apiserver kube-scheduler kube-controller-manager" --build-arg VERSION=1.30.4 --build-arg SOURCE_DATE_EPOCH=1724341750 --build-arg BUILDIMAGE=docker.io/library/golang:1.22.5-alpine3.20 --build-arg BUILD_GO_TAGS="providerless" --build-arg BUILD_GO_CGO_ENABLED= --build-arg BUILD_GO_CGO_CFLAGS='' --build-arg BUILD_SHIM_GO_CGO_ENABLED= --build-arg BUILD_GO_FLAGS="-v" --build-arg BUILD_GO_LDFLAGS= --build-arg BUILD_GO_LDFLAGS_EXTRA="-extldflags=-static" -- kubernetes/

That said, this is impressively close now! I will look into this more, chances are the sought kubelet bin got put somewhere else unexpectedly? Either way, just wanted to drop that update here for the time being. :)

[IngwiePhoenix]

@twz123 I am going to try your command next.

When running, it immediately says that riscv64 is unknown and it will assume amd64. It also errors out at the same step in the build. For reference:

# git rev-parse HEAD
5db0100a02e6ece8fd6dbbe40a76a5e44aa49118

I couldn't find the patches that were mentioned though. That said, you did mention in that other ticket that currently it is not possible to build off of a clean clone. Do you still have the patches you made to the build at hand somewhere?