Add support for `riscv64` architecture

[MengMengDaXiaoJi]

edited by @brandond Repurposing this as a tracking issue for riscv64 support.

This has not been prioritized, and we don't have any build infra for riscv64, so at the moment this will be purely experimental.

[dominch]

Recently ubuntu start to ship docker ready image for starfive vision five 2 with kernel already configured for docker and kubernetes:

root@ubuntu:/home/ubuntu# curl https://raw.githubusercontent.com/k3s-io/k3s/master/contrib/util/check-config.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13993  100 13993    0     0  44061      0 --:--:-- --:--:-- --:--:-- 44141

Verifying binaries in .:
- sha256sum: sha256sums unavailable
- links: link list unavailable

System:
- /usr/sbin iptables v1.8.7 (nf_tables): ok
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/6.2.0-19-generic
info: reading kernel config from /boot/config-6.2.0-19-generic ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: pass

It's huge change, earlier custom built kernel was needed to get at least docker/containers to work. For now boards with already updated u-boot just get ubuntu official support with quite good kernel. Still no official support:

root@ubuntu:/home/ubuntu# curl -sfL https://get.k3s.io | sh -
[ERROR]  Unsupported architecture riscv64

[brandond]

Yeah we don't have any hardware for CI or QA so we probably won't have official support any time soon.

[dominch]

RiscV is getting more and more popular and AFAIK servers with this arch are already on the way. After very successful kickstarter for star five vision five 2 board still is available on few e-shops for <$100. Of course it's still dev board and there are some issues to fix, but it's first accessible platform with some power. Also I'm sure that You need something more professional than this ;) Of course You don't need such hardware to test that, QEMU is able to emulate at least older RiscV - Vision Five board. I don't know quality of such emulation, but it should be better than usual and probably You will get better results on moderate x86 than on original board (which is rather slow on hardware level). If everything fails and You are willing to get access to such board let me know, I have two with working eMMC, and partially nvme. I can setup one of them on cloud for long time. Also alibaba cloud is preparing something online too, but for now I could not find details.

[okias]

Would it help if I would provide access to the 3x VisionFive V2?

[brandond]

Our team already has access to dev hardware, however we don't yet have access to colocated servers that could reliably build for this platform. Until then, we are limited to cross-building, which is slow - and we're reluctant to add that to our pipeline at this time.

[brandond]

Just for the record:

unmatched01:~ # kubectl get node -o wide
NAME                    STATUS   ROLES                       AGE   VERSION                INTERNAL-IP   EXTERNAL-IP   OS-IMAGE              KERNEL-VERSION     CONTAINER-RUNTIME
unmatched01.lan.khaus   Ready    control-plane,etcd,master   27m   v1.28.1+k3s-b44521db   10.0.1.182    <none>        openSUSE Tumbleweed   6.4.12-1-default   containerd://1.7.3-k3s2
unmatched02.lan.khaus   Ready    control-plane,etcd,master   19m   v1.28.1+k3s-b44521db   10.0.1.212    <none>        openSUSE Tumbleweed   6.4.12-1-default   containerd://1.7.3-k3s2
unmatched03.lan.khaus   Ready    control-plane,etcd,master   14m   v1.28.1+k3s-b44521db   10.0.1.188    <none>        openSUSE Tumbleweed   6.4.12-1-default   containerd://1.7.3-k3s2

[brandond]

Note that there are no images available to run on these nodes yet, as our rancher-mirrored images exclude the riscv platform.

[dominch]

Thanks for update! when do You think that those images will be available? Do You plan any beta/alpha version?

[brandond]

You can follow https://github.com/rancher/image-mirror/issues/502

[plsnotracking]

Hello, I'm here to just mention, I have a RISC-V 64 (StarFive 2) machine as well, and would like to add it my existing k3s cluster. Thank you for the work, all the best.

[pascal71]

For those interested; I have K3S running on 4 StarFive Vision 2s using https://github.com/CARV-ICS-FORTH/kubernetes-riscv64. I have been able to port some more containers like the demo programs I use in my K8S classes as well as NFS-STORAGE-PROVISIONER. I am currently studying porting OpenEBS and Longhorn.

It looks like the RISCV64 for K3S porting project is stalled a little bit, so I am trying to fork my own tree for experiments. Biggest 'next' challenge is the distroless container images everyone has migrated too. (Where are the Dockerfiles? :) ) RISCV64 now being a fully supported target and hostplatform on Go makes the experience much more pleasant than a year before. :)

[brandond]

We're kind of just waiting for the ecosystem to catch up. Once upstreams for all our various helper projects, images, and build tools support riscv64 this will be easier to move forward.

[omnivagant]

In alpinelinux, we have packaged k3s for riscv64 in our rolling release "edge" for a couple of years. The latest release v1.29.1+k3s1, however, broke the build by pulling in github.com/marten-seemann/tcp since #8977 so we had to disable k3s for riscv64 again, see also https://github.com/marten-seemann/tcp/pull/1

[brandond]

Oh, that's fun.

[IngwiePhoenix]

We're kind of just waiting for the ecosystem to catch up. Once upstreams for all our various helper projects, images, and build tools support riscv64 this will be easier to move forward.

Which ones, exactly?

I have a stable VisionFive2 as well and did a blind attempt at building, only to be greeted by Dapper not knowing what to do. What are the core components? I tried tracing the Makefile and scripts to find out precisely which components are needed to build k3s. Do you happen to have something like a "dependency list" in this regard? With it, I could try and see which of them do build on riscv64 and work, and later attempt to help upstream potential fixes.

As for Go itself, you should be able to install [gcc,g++}-linux-gnu-riscv64 and then just set GOARCH - to enable CGO support... since I am testing most things directly on the board, I am not completely sure. But basically all the things built with just Go itself have worked so far, and very reliably too, no less (i.e. syncthing, rclone).

Would love to help!

[IngwiePhoenix]

So I gave this another spin and took a look at all the things that are required during build.

And the biggest one is: Dapper needs containers - to be specific, Dockerfile.dapper specifies golang:1.21.6-alpine3.18. Whilst there are a few riscv64 images (busybox, some alpine images, debian, ubuntu), those very specific tags do not exist.

Is there a way to build without Dapper? From what I could gather, all it really does is set up a Go environment in the container and then run arbitrary commands there - and it also seems to use Trivy, but I haven't yet figured out how or where.

Dapper built just fine with go install github.com/rancher/dapper@latest, by the way. Haven't tried trivy yet.

For porting purposes, it'd be nice to have a build script, or instructions, on how to build without needing to rely on Dapper.

In the meantime, I will keep trying. :) BTW, I set the repo to v1.29.1+k3s2. Long term I want to match k3s versions across my cluster - and then there is also the problem of building non-existing images and tagging them in a way that k3s will find them...

[pascal71]

Quoting Ingwie Phoenix @.***>:

Good day Ingwie,

Kindly check https://github.com/CARV-ICS-FORTH/k3s/releases

It is (was) a sponsored project. I am working with it for 3 months
now, getting it working on Starfive Vision5-2 SOC. 'till January 31th
it wasn't working on RISCV64 Licheepi4a due to a kernel misconfig. I
can confirm it's working again. This project most probably was used as
more of a Proof Of Concept. They support 1.27.3 of K8S/K3S and no new
releases have been created after this.

If you'd like we can work together to make it work for K8S 1.29.x and
higher. Any help/support/advise from Rancher/SUSE/K3S would be helpful.

If you have further questions, kindly let me know.

Kind regards,

Pascal van Dam

So I gave this another spin and took a look at all the things that
are required during build.

And the biggest one is: Dapper needs containers - to be specific,
Dockerfile.dapper specifies golang:1.21.6-alpine3.18. Whilst there are a few riscv64 images (busybox, some alpine images,
debian, ubuntu), those very specific tags do not exist.

Is there a way to build without Dapper? From what I could gather,
all it really does is set up a Go environment in the container and
then run arbitrary commands there - and it also seems to use Trivy,
but I haven't yet figured out how or where.

Dapper built just fine with go install ***@***.***, by the way. Haven't tried trivy
yet.

For porting purposes, it'd be nice to have a build script, or
instructions, on how to build without needing to rely on Dapper.

In the meantime, I will keep trying. :) BTW, I set the repo to
v1.29.1+k3s2. Long term I want to match k3s versions across my
cluster - and then there is also the problem of building
non-existing images and tagging them in a way that k3s will find
them...

-- Reply to this email directly or view it on GitHub: https://github.com/k3s-io/k3s/issues/7151#issuecomment-1951689668 You are receiving this because you commented.

Message ID: @.***>

[IngwiePhoenix]

Hello @pascal71 ! I saw that repo, and in fact, that is the release I currently run. But, I wanted to see what I could do to build the upstream version as well.

As it turns out, k3s-root has a riscv root now since v0.13.0 - which is highly promising!

Here are a few changes I did in order to try and make it work:

diff --git a/Makefile b/Makefile
index 620f899894..b82c76dc0d 100644
--- a/Makefile
+++ b/Makefile
@@ -4,8 +4,9 @@ GO_FILES ?= $$(find . -name '*.go' | grep -v generated)

 .dapper:
        @echo Downloading dapper
-       @curl -sL https://releases.rancher.com/dapper/v0.6.0/dapper-$$(uname -s)-$$(uname -m) > .dapper.tmp
-       @@chmod +x .dapper.tmp
+       #@curl -sL https://releases.rancher.com/dapper/v0.6.0/dapper-$$(uname -s)-$$(uname -m) > .dapper.tmp
+       #@@chmod +x .dapper.tmp
+       #@ln -s $$(which dapper) .dapper.tmp
        @./.dapper.tmp -v
        @mv .dapper.tmp .dapper

@@ -42,4 +43,4 @@ format:
 local:
        DOCKER_BUILDKIT=1 docker build \
                --build-arg="REPO TAG GITHUB_TOKEN GOLANG GOCOVER DEBUG" \
-               -t k3s-local -f Dockerfile.local --output=. .
\ No newline at end of file
+               -t k3s-local -f Dockerfile.local --output=. .
diff --git a/scripts/version.sh b/scripts/version.sh
index f7bc93fef8..228b84591c 100755
--- a/scripts/version.sh
+++ b/scripts/version.sh
@@ -75,7 +75,7 @@ if [ -z "$VERSION_KUBE_ROUTER" ]; then
     VERSION_KUBE_ROUTER="v0.0.0"
 fi

-VERSION_ROOT="v0.12.2"
+VERSION_ROOT="v0.13.0"

 DEPENDENCIES_URL="https://raw.githubusercontent.com/kubernetes/kubernetes/${VERSION_K8S}/build/dependencies.yaml"
 VERSION_GOLANG="go"$(curl -sL "${DEPENDENCIES_URL}" | yq e '.dependencies[] | select(.name == "golang: upstream version").version' -)

But in the end, ignoring Makefile entirely and just using the scripts/ directory was the better way to go. I am actually looking into how to properly update the one last module blocking a build:

# github.com/marten-seemann/tcp
/root/go/pkg/mod/github.com/marten-seemann/tcp@v0.0.0-20210406111302-dfbc87cc63fd/sys_linux.go:8:19: undefined: sysSIOCINQ
/root/go/pkg/mod/github.com/marten-seemann/tcp@v0.0.0-20210406111302-dfbc87cc63fd/sys_linux.go:9:19: undefined: sysSIOCOUTQ

This was actually fixed a while later. But I am quite new to Go development... so, I am still looking at how to efficiently update that specific module. After that, the build should pass.

mkdir -p build/data
# Make sure to edit the k3s-root version reference.
./scripts/download
./scripts/generate
# This is where I am stuck.
./scripts/build

It's quite close and the fixes required are minimal. That said, I am still on an older branch (v1.29.1+k3s2), chances are this has been edited since, but I haven't had the time to crawl through the commit history just yet.

Do you happen to know how I can edit that specific dependency?

Thanks, and kind regards, Ingwie

[brandond]

I think I said this earlier, but it is not hard to build k3s for riscv64. The issue is that none of our images currently support riscv64. Up until recently alpine didn't support it either, except in edge. Once we actually have something to run on this architecture we will reevaluate making binaries available.

[IngwiePhoenix]

So do I understand you correctly that all you need is a set of Alpine images compiled for RISC-V? I have no problem letting my SF2 run overtime and compile some rootfs's - I just installed a 1TB NVMe SSD into it. I'd be happy to donate my spare compute if this helps others in the process! Let me know what you need precisely, and I will see if I can find the people responsible for https://hub.docker.com/u/riscv64 and figure something out with them. All my SF2 does is TVHeadend and a little bit of Home Assistant - so most of the time, it is idle anyway. :)

[IngwiePhoenix]

I am down to two problems: Setting rootfs in scripts/version.sh to 0.13.0 and now trying to monkey-patch that one damn dependency that snuck in...

There is apparently a fix for that here in this PR: https://github.com/marten-seemann/tcp/pull/1#issuecomment-1742126490 So... Temporarily, I learned I can use replace:

replace github.com/marten-seemann/tcp => github.com/r-value/tcp v0.0.0-20211124094235-5d06fda83888

This results in this:

diff --git a/go.mod b/go.mod
index c8f1d243ff..167f318209 100644
--- a/go.mod
+++ b/go.mod
@@ -2,6 +2,8 @@ module github.com/k3s-io/k3s

 go 1.21

+replace github.com/marten-seemann/tcp => github.com/r-value/tcp v0.0.0-20211124094235-5d06fda83888
+
 replace (
        github.com/Microsoft/hcsshim => github.com/Microsoft/hcsshim v0.11.0
        github.com/Mirantis/cri-dockerd => github.com/k3s-io/cri-dockerd v0.3.9-k3s2 // k3s/release-1.28
diff --git a/go.sum b/go.sum
index 858cd6e414..88d4a49714 100644
--- a/go.sum
+++ b/go.sum
@@ -1140,8 +1140,6 @@ github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czP
 github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
 github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
 github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
-github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd h1:br0buuQ854V8u83wA0rVZ8ttrq5CpaPZdvrK0LP2lOk=
-github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd/go.mod h1:QuCEs1Nt24+FYQEqAAncTDPJIuGs+LxK1MCiFL25pMU=
 github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
 github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
 github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
@@ -1475,6 +1473,8 @@ github.com/quic-go/quic-go v0.38.2 h1:VWv/6gxIoB8hROQJhx1JEyiegsUQ+zMN3em3kynTGd
 github.com/quic-go/quic-go v0.38.2/go.mod h1:ijnZM7JsFIkp4cRyjxJNIzdSfCLmUMg9wdyhGmg+SN4=
 github.com/quic-go/webtransport-go v0.5.3 h1:5XMlzemqB4qmOlgIus5zB45AcZ2kCgCy2EptUrfOPWU=
 github.com/quic-go/webtransport-go v0.5.3/go.mod h1:OhmmgJIzTTqXK5xvtuX0oBpLV2GkLWNDA+UeTGJXErU=
+github.com/r-value/tcp v0.0.0-20211124094235-5d06fda83888 h1:skZUklgonyd2bz0llv7AjdCuaTeZ1JP5hSqQi5m9aE4=
+github.com/r-value/tcp v0.0.0-20211124094235-5d06fda83888/go.mod h1:QuCEs1Nt24+FYQEqAAncTDPJIuGs+LxK1MCiFL25pMU=
 github.com/rancher/dynamiclistener v0.3.6 h1:iAFWeiFNra6tYlt4k+jINrK3hOxZ8mjW2S/9nA6sxKs=
 github.com/rancher/dynamiclistener v0.3.6/go.mod h1:VqBaJNi+bZmre0+gi+2Jb6jbn7ovHzRueW+M7QhVKsk=
 github.com/rancher/lasso v0.0.0-20230830164424-d684fdeb6f29 h1:+kige/h8/LnzWgPjB5NUIHz/pWiW/lFpqcTUkN5uulY=
diff --git a/scripts/version.sh b/scripts/version.sh
index a67f0d5607..939465c0f2 100755
--- a/scripts/version.sh
+++ b/scripts/version.sh
@@ -75,7 +75,7 @@ if [ -z "$VERSION_KUBE_ROUTER" ]; then
     VERSION_KUBE_ROUTER="v0.0.0"
 fi

-VERSION_ROOT="v0.12.2"
+VERSION_ROOT="v0.13.0"

 DEPENDENCIES_URL="https://raw.githubusercontent.com/kubernetes/kubernetes/${VERSION_K8S}/build/dependencies.yaml"
 VERSION_GOLANG="go"$(curl -sL "${DEPENDENCIES_URL}" | yq e '.dependencies[] | select(.name == "golang: upstream version").version' -)

Still building since a good 30min. This'll take a while, as I am doing this on the target. But from what all I could tell, this is literally all that is needed.

By the way, neither of those .../tcp packages is maintained anymore, at all. They've been dormant for very long now; but some packages seem to still use that?

Is using replace a viable strategy here?

[IngwiePhoenix]

Success.

build/out/data.tar.zst (scripts/package failed because of the missing alpine:3.19 container - but, it got this far, which is enough.)

[plsnotracking]

@IngwiePhoenix do you happen to have a final set of instructions to build this locally, and have it run? Thank you for doing all the recon work, and making it work.

[IngwiePhoenix]

@plsnotracking Sure - it's actually really simple ... once you spent hours figuring it out, learning some Go in the meantime and getting to know a bunch of k3s repos. x)

1. git clone -b "v1.29.2+k3s1" https://github.com/k3s-io/k3s.git
2. Edit scripts/version.sh
   • Find: VERSION_ROOT="v0.12.2"
   • Replace with: VERSION_ROOT="v0.13.0"
3. Edit go.mod
   • Find the line starting with: go 1.21...
   • Beneath that, add: replace github.com/marten-seemann/tcp => github.com/r-value/tcp latest
4. Run go mod tidy
   • The entry in go.mod should have changed latest to a proper version string now - helps you verify things worked as expected!
5. Run scripts/download
6. Then scripts/generate
7. And finally, scripts/build
8. Lastly, scripts/package-cli
   • The other ones require container images to build, which do not exist for RISC-V ... yet? ;) Working on that atm.

You now have a fully built k3s! The patch itself is super small, and you just have to ignore the fancy containerized build start to finish - but other than that, this works. It took me way too long to figure this out...

Give it a try! The build doesn't actually take this long - building Vaultwarden took much longer. Also, my final repo size is:

# du -d 0 -h k3s
1.4G    k3s

About half of a built Linux kernel tree - so you should easily be able to build this on an eMMC or something. Throw it into a screen and make a sandwhich. :)

[plsnotracking]

Thank you.

Although tangential, I had a small problem running into cloning the repo

❯ git clone -b "v1.29.2+k3s1" https://github.com/k3s-io/k3s.git
Cloning into 'k3s'...
remote: Enumerating objects: 994636, done.
remote: Counting objects: 100% (1896/1896), done.
remote: Compressing objects: 100% (897/897), done.
error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8)
error: 7160 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

I will try to increase the buffer size or fallback to http 1.1 which seems backwards, but I guess, that's a solution.

[IngwiePhoenix]

Try the SSH method (git@github.com:k3s-io/k3s.git). Have to do the same whenever I clone the Star-Tech JH7110 kernel. Also, do you have git LFS installed?

[AllardKrings]

Hi,

if someone is looking for some riscv64 images, look in my repository on docker hub: allardkrings/riscv64….

[IngwiePhoenix]

I have no idea what, how, or when, but using just the shell scripts and ignoring Dapper, k3s builds perfectly fine now!

root@riscboi /n/o/k3s ((v1.30.4+k3s1))# ./scripts/download; and ./scripts/build; and ./scripts/package-cli            
# ... build ...
root@riscboi /n/o/k3s ((v1.30.4+k3s1))# dist/artifacts/k3s-riscv64  --version
k3s-riscv64 version v1.30.4+k3s1 (98262b5d)
go version go1.22.6

Doesn't even have the -dirty tag anymore since the repo is entirely unmodified.

Guess the next folks I have to bother is the Dapper devs. =) But this is already pretty amazing. I will try to create an exemplary cluster in my spare time on the weekend to see if it works and how far that goes - but it is getting there!

[macabu]

I was able to build it and run without any modifications as well:

matheus@bananapif3:~/projects/k3s-1.31.0-k3s1/dist/artifacts$ file k3s-riscv64
k3s-riscv64: ELF 64-bit LSB executable, UCB RISC-V, double-float ABI, version 1 (SYSV), statically linked, Go BuildID=CNTMeyK6zfqNoG_8tf9Y/VeWQpkv5wHO-CF7fbVKk/AS2IhtUPexyrtYdAcPZw/w9_GGPikbHqd-UOkmAKk, stripped
matheus@bananapif3:~/projects/k3s-1.31.0-k3s1/dist/artifacts$ ldd k3s-riscv64
    not a dynamic executable
matheus@bananapif3:~/projects/k3s-1.31.0-k3s1/dist/artifacts$ ./k3s-riscv64 --version
k3s-riscv64 version v1.31.0+k3s- ()
go version go1.22.5

But as you've pointed out there are some images that are not available for riscv64, and I got this error:

E0916 13:01:04.771972   44618 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": no match for platform in manifest: not found"

I've found that the pause image is being mirrored from Kubernetes directly

[IngwiePhoenix]

I do believe there is a way to configure those images to be sourced from alternative paths?

Did you build with the scripts or with Dapper?

[macabu]

Just with the scripts, thanks for the instructions! I am able to build the pause image myself, but for the upstreaming efforts it looks a bit more complicated 😬

There's some requirement on kube-cross to build the pause image, and kube-cross in turn depends on a Go + Debian image. AFAIK, Debian only added support for RISCV64 recently with Trixie, which won't become stable for at least another year 😅

[IngwiePhoenix]

Yep, that's correct. However: The definition OS_CODENAME ?= bullseye means that you can overwrite it.

env OS_CODENAME=trixie make...

So, technically it's possible. That said, I don't know where/when this is built or not. Oh well, I've been waiting this long, I can wait a little longer too! But seriously there should be a configuration for that. I very much remember having that recommended in either another issue related to riscv support or in a linked repository...darn, I can't remember it. :/

also you're welcome!

[macabu]

I think the main hurdle is needing to build a custom image of golang:1.22.6-bullseye (trixie) in that case. It's doable of course, but anyway the K8s team is going to wait for an official golang image before updating it