k3s as RPMs

[SchoolGuy]

Is your feature request related to a problem? Please describe.

At the moment is is not possible to build k3s as an RPM or DEB package. This means I have to use the curl installer to install k3s on my host system. For me this represents an inconvenience because I would like to manage my system(s) via RPM/DEB packages.

Describe the solution you'd like

A build process which would build k3s only from sources checked in before build time. This would enable any Linux distribution to ship k3s in their repositories which should increase the visibility in a great manner. Specific build chains like dapper should not be involved in that process.

Describe alternatives you've considered

I have tried my best to understand how the current build system works and it is not even possible to exchange docker with podman without further bash reconfiguration (symlinks etc.). I have tried to understand the necessity of this - imho - inflexible buildchain and I lack the understanding of the decision sadly. I have tried to research the reasons but I was not successful.

Having a consistent environment for building can be achieved in many ways, I don't see a reason why this has to be achieved with dapper and continuos internet access.

Additional context

You said you'd disable this "for now" and reevaluate this later in #2440. I hope that this reevaluation can be done now. Obviously #1535 would reopen which I would also take care of then.

I am willing to invest my time to maintain this solution in the long term if we can find an agreement on how it should look like according to my desired solution.

Having RPMs and DEBs is not that hard if we consider the following:

• The specific requirements, for example for runc or containerd, can be built as subpackages of k3s. The build process is already available in most distributions, meaning that a git clone ... like done here is not required.
• The Traefik Helmchart can easily be built in any environment if the sources are previously downloaded since there is no black magic involved for this part of the build.
• For the userspace tools like iptables which are currently a part of the k3s-root repository one can easily again just branch the required version from the distribution of choice like the first bullet point already describes.

The effort of doing this should be relatively small since the packaging of all the vendored dependencies is already done, they just need to be taken at the desired point in time. If architected well this doesn't even mean dapper is discontinued but in the end it would be just one options/the preferred option to build k3s.

For me the strongest point for such a solution is that the trust in k3s should be much higher when it comes from the vendor of your OS instead of a third party. Because normally packages are maintained by your OS vendor it is easy to trust them. If I execute a script of a third party I normally would need to establish trust to such a party first.

Additionally if I have an RPM/DEB I have a defined version which is installed. This means I know that a specific version will install a set of files on my system to previously known locations. When I install k3s via the current curl method I don't know what, where and how k3s will install itself on my system.

Lastly if an RPM would be available, one could take advantage of things like the SUSE Manager/Uyuni Content lifecycle management which enables automated staging of the k3s RPMs. This means a company can have an automated rollout for their k3s package with the tools already present in an environment, instead of a custom tailored solution.

Example: One could have a staging cluster where a new k3s package (with a bugfix or new version) is deployed to. If that is successful the package would be promoted to the next environment where the likelihood of that failing is much less then if I would install this directly in an environment.

Backporting

• [ ] Needs backporting to older releases

[SchoolGuy]

Okay I want to provide a tldr as a question: What are the requirements for you that k3s can be built in the OBS as an RPM?

It is not my intention to complicate things with this request but to make it easier. For me installing an RPM is the easiest way to manage the software I have on my system. I understand that this is not the opinion of everyone but I would like to see it possible that I can go my way and others can go theirs.

[SchoolGuy]

Okay I am running into build trouble while building k3s but so far the changes I have come up with are rather minimal...

diff --git a/Makefile b/Makefile
index d18f7c1884..c1682be0ea 100644
--- a/Makefile
+++ b/Makefile
@@ -7,8 +7,10 @@ TARGETS := $(shell ls scripts | grep -v \\.sh)
        @./.dapper.tmp -v
        @mv .dapper.tmp .dapper

+ifndef SKIP_DAPPER
 $(TARGETS): .dapper
        ./.dapper $@
+endif

 .PHONY: deps
 deps:
@@ -32,3 +34,7 @@ binary-size-check:
 .PHONY: image-scan
 image-scan:
        scripts/image_scan.sh $(IMAGE)
+
+.PHONY: source-tarball
+source-tarball: deps
+       ./scripts/download
diff --git a/scripts/build b/scripts/build
index d8c27c1400..7d5b4138d2 100755
--- a/scripts/build
+++ b/scripts/build
@@ -97,17 +97,22 @@ cleanup() {
     exit ${exit_status}
 }

-INSTALLBIN=$(pwd)/bin
-if [ ! -x ${INSTALLBIN}/cni ]; then
-(
-    echo Building cni
-    TMPDIR=$(mktemp -d)
-    trap cleanup EXIT
-    WORKDIR=$TMPDIR/src/github.com/containernetworking/plugins
-    git clone -b $VERSION_CNIPLUGINS https://github.com/rancher/plugins.git $WORKDIR
-    cd $WORKDIR
-    GO111MODULE=off GOPATH=$TMPDIR CGO_ENABLED=0 "${GO}" build -tags "$TAGS" -ldflags "$VERSIONFLAGS $LDFLAGS $STATIC" -o $INSTALLBIN/cni
-)
+if [ -z "$SKIP_CNI" ]; then
+    INSTALLBIN=$(pwd)/bin
+    if [ ! -x ${INSTALLBIN}/cni ]; then
+    (
+        echo Building cni
+        TMPDIR=$(mktemp -d)
+        trap cleanup EXIT
+        WORKDIR=$TMPDIR/src/github.com/containernetworking/plugins
+        git clone -b $VERSION_CNIPLUGINS https://github.com/rancher/plugins.git $WORKDIR
+        cd $WORKDIR
+        GO111MODULE=off GOPATH=$TMPDIR CGO_ENABLED=0 "${GO}" build -tags "$TAGS" -ldflags "$VERSIONFLAGS $LDFLAGS $STATIC" -o $INSTALLBIN/cni
+    )
+    fi
+else
+    # TODO: Use pre-built binary if available
+    echo "cni: Using previusly built binary if available"
 fi

 echo Building k3s
diff --git a/scripts/package b/scripts/package
index 51b8f417bb..ca2123657e 100755
--- a/scripts/package
+++ b/scripts/package
@@ -8,7 +8,9 @@ if [ ! -e ../bin/containerd ]; then
 fi

 ./package-cli
-./package-image
+if [ -z "$SKIP_IMAGE" ]; then
+    ./package-image
+fi
 if [ -z "$SKIP_AIRGAP" ]; then
     ./package-airgap
 fi

[SchoolGuy]

Okay I tried digging further into the build failure but I don't find a clue here to continue sadly. Help would be much appreciated:

Building k3s
+ CGO_ENABLED=1
+ go build -v -tags 'apparmor seccomp netcgo osusergo providerless' -ldflags '
    -X github.com/k3s-io/k3s/pkg/version.Version=v1.24.1-rc1+k3s1
    -X github.com/k3s-io/k3s/pkg/version.GitCommit=a5a0e8fd

    -X k8s.io/client-go/pkg/version.gitVersion=v1.24.1-rc1+k3s1
    -X k8s.io/client-go/pkg/version.gitCommit=a5a0e8fde2dd3e75b12bc9c22a1bceb74d957025
    -X k8s.io/client-go/pkg/version.gitTreeState=dirty
    -X k8s.io/client-go/pkg/version.buildDate=2022-05-29T12:08:45Z

    -X k8s.io/component-base/version.gitVersion=v1.24.1-rc1+k3s1
    -X k8s.io/component-base/version.gitCommit=a5a0e8fde2dd3e75b12bc9c22a1bceb74d957025
    -X k8s.io/component-base/version.gitTreeState=dirty
    -X k8s.io/component-base/version.buildDate=2022-05-29T12:08:45Z

    -X github.com/kubernetes-sigs/cri-tools/pkg/version.Version=v1.24.0-k3s1

    -X github.com/containerd/containerd/version.Version=v1.6.4-k3s1
    -X github.com/containerd/containerd/version.Package=github.com/k3s-io/containerd

    -X github.com/containernetworking/plugins/pkg/utils/buildversion.BuildVersion=v1.1.1-k3s1
    -X github.com/containernetworking/plugins/plugins/meta/flannel.Program=flannel
    -X github.com/containernetworking/plugins/plugins/meta/flannel.Version=v0.17.0
    -X github.com/containernetworking/plugins/plugins/meta/flannel.Commit=a5a0e8fde2dd3e75b12bc9c22a1bceb74d957025
    -X github.com/containernetworking/plugins/plugins/meta/flannel.buildDate=2022-05-29T12:08:45Z

    -w -s 
' -o bin/k3s ./cmd/server/main.go
pkg/agent/run.go:15:2: //go:build comment without // +build comment
pkg/agent/proxy/apiproxy.go:11:2: //go:build comment without // +build comment
pkg/agent/run.go:19:2: //go:build comment without // +build comment
pkg/agent/containerd/config_linux.go:13:2: //go:build comment without // +build comment
pkg/agent/config/config.go:24:2: //go:build comment without // +build comment
pkg/agent/config/config.go:26:2: //go:build comment without // +build comment
pkg/server/context.go:10:2: //go:build comment without // +build comment
pkg/agent/run.go:29:2: //go:build comment without // +build comment
pkg/server/server.go:27:2: //go:build comment without // +build comment
pkg/daemons/config/types.go:13:2: //go:build comment without // +build comment
abuild@tower:~/rpmbuild/BUILD/k3s> echo $?
1

[SchoolGuy]

Okay the upgrade to go 1.17 solved my issue. It seems the go.mod file on master is maybe outdated?

[SchoolGuy]

I was successful 🥳

[   88s] RPMLINT report:
[   88s] ===============
[   94s] ============================ rpmlint session starts ============================
[   94s] rpmlint: 2.3.0
[   94s] configuration:
[   94s]     /opt/testing/lib64/python3.10/rpmlint/configdefaults.toml
[   94s]     /opt/testing/share/rpmlint/cron-whitelist.toml
[   94s]     /opt/testing/share/rpmlint/dbus-services.toml
[   94s]     /opt/testing/share/rpmlint/device-files-whitelist.toml
[   94s]     /opt/testing/share/rpmlint/licenses.toml
[   94s]     /opt/testing/share/rpmlint/opensuse.toml
[   94s]     /opt/testing/share/rpmlint/pam-modules.toml
[   94s]     /opt/testing/share/rpmlint/permissions-whitelist.toml
[   94s]     /opt/testing/share/rpmlint/pie-executables.toml
[   94s]     /opt/testing/share/rpmlint/polkit-rules-whitelist.toml
[   94s]     /opt/testing/share/rpmlint/scoring.toml
[   94s]     /opt/testing/share/rpmlint/security.toml
[   94s]     /opt/testing/share/rpmlint/sudoers-whitelist.toml
[   94s]     /opt/testing/share/rpmlint/systemd-tmpfiles.toml
[   94s]     /opt/testing/share/rpmlint/users-groups.toml
[   94s]     /opt/testing/share/rpmlint/world-writable-whitelist.toml
[   94s] checks: 40, packages: 2
[   94s] 
[   94s] k3s.x86_64: E: update-alternatives-requirement-missing
[   94s] The package does not have update-alternatives in Requires(post) or
[   94s] Requires(postun). This is needed for the proper scriptlet execution.
[   94s] 
[   94s] k3s.x86_64: E: update-alternatives-post-call-missing
[   94s] The package does not call update-alternatives --install in post phase to
[   94s] install all the configuration.
[   94s] 
[   94s] k3s.x86_64: E: statically-linked-binary /usr/bin/k3s
[   94s] The package installs a statically linked binary or object file.
[   94s] 
[   94s] k3s.x86_64: W: position-independent-executable-suggested /usr/bin/k3s
[   94s] This executable should be position independent (all binaries should).  Check
[   94s] that it is built with -fPIE/-fpie in compiler flags and -pie in linker flags.
[   94s] 
[   94s] k3s.x86_64: I: package supports update-alternatives
[   94s] k3s.x86_64: W: non-executable-in-bin /usr/bin/k3s 644
[   94s] A file is being installed in /usr/bin, but is not an executable. Be sure that
[   94s] the file is an executable or that it has executable permissions.
[   94s] 
[   94s] Check time report (>1% & >0.1s):
[   94s]     Check                            Duration (in s)   Fraction (in %)  Checked files
[   94s]     rpm2cpio                                     3.3              62.8               
[   94s]     SignatureCheck                               1.8              34.4               
[   94s]     TOTAL                                        5.2             100.0             17
[   94s] 
[   94s]  2 packages and 0 specfiles checked; 3 errors, 2 warnings, 3 badness; has taken 5.3 s 
[   94s] 
[   94s] 
[   94s] tower.fritz.box finished "build k3s.spec" at Mon May 30 19:56:22 UTC 2022.
[   94s] 

/var/tmp/build-root/openSUSE_Tumbleweed-x86_64/home/abuild/rpmbuild/SRPMS/k3s-1.23.5+k3s1-0.src.rpm

/var/tmp/build-root/openSUSE_Tumbleweed-x86_64/home/abuild/rpmbuild/RPMS/x86_64/k3s-1.23.5+k3s1-0.x86_64.rpm

A lot can be improved by me optimizing the specfile and ripping the build in multiple subpackages so eventually one can build with a _service file and there is no need anymore to execute something on a local machine.

[SchoolGuy]

@brandond What would be your wishes in terms for my refactoring of the k3s build scripts? I have multiple ideas and options. The minimal intrusive changes I already posted above but for a better compatibility I would love to do more. I have much more ideas for cleaning up the scripts folder. I would love to upstream those ideas and even maintain the rpm for you in openSUSE. The only thing I would need is that we work together in terms of what you need so I can find the middle ground that suits both of us.

[TheBigBear]

@SchoolGuy do you have a git branch that allows one to build [ pre-release and unofficial ( for now)] k3s RPMS right now?

[SchoolGuy]

@TheBigBear I have something locally. I will clean it up and then provide a branch plus PR. Will take me over the weekend probably.

[SchoolGuy]

@TheBigBear I have created WIP PR #5793, I will try to take time and refine it during the weekend.

[cwayne18]

While we're happy for the k3s community to support/maintain k3s as RPM's, there is currently not any plans for the k3s team to build on OBS as an upstream.

[SchoolGuy]

@cwayne18 Yes I am aware and accept that. However it would be nice of your buildscripts to make it easier and allow it. I have a WIP PR open where I am currently refactoring them so it is easier to do that.

[benridley]

@cwayne18 I would love for the k3s team to reconsider their position on not providing official RPM/DEB packages.

If I can share my experience: Installing Kubernetes in enterprise environments is often a horrendous experience with a multitude of random shell scripts, container images, systemd units, CNI plugins, etc all required. This is especially bad in air-gapped environments where you must bring all these things together on some internally hosted infra. In my opinion, k3s is uniquely placed to address these challenges because it is already delivered as a single binary. The installation process is really lacking, however, because it relies on running a shell script that brings in lots of artifacts from across the internet and does not play nicely with the native package managers that exist on RHEL and Debian systems.

k3s would be a no-brainer in enterprise environments if the team provided official RPM and Deb repos that allowed organisations and administrators to simply dnf/apt install k3s and get everything you needed. Plus, this provides a simple and sustainable way for admins to update k3s because they can utilize the existing package management mechanisms. Right now, this is also a manual process relying on downloading and unpacking releases etc.

I would be happy to provide effort to enable this. I think it would be the best way to deploy and maintain k8s in these kinds of environments.

[SchoolGuy]

@benridley My PR is doing 90% of the work for this. Feel free to test it. I am not able to get SELinux to work atm because either I need to adjust the SELinux policy to take the host binaries or I need to link the userspace of whatever OS I am building on atm. Both things are the famous 10% that take 80% of the time. If you don't care about SELinux my PR is working wonderfully according to my testing.

[UnconventionalMindset]

is anyone still working or interested in this?

[SchoolGuy]

I am. However I am blocked by the k3s maintainers to actually get this in.

Edit: My RPM in the OBS is not compatible with selinux due to the missing copy of the userspace. However without selinux the RPM that is depending on my PR is fully functional.

[c4t3l]

I am also interested in this. Utilizing rpm for this purpose just makes plain sense in my corporate environment. Not only does it help with air-gapped installations, it also provides an audit trail via dnf and rpmdb that can be queried by third party security tools etc.

[jdptechnc]

I would also be interested in this.

[SchoolGuy]

As soon as I have a green light from the k3s team I will rebase my PR and rebuild the RPM that is in my home project to show that the changes I made are working as intended. I am willing to commit to packaging different release series in OBS for the community in a dedicated OBS project if the k3s team is interested. I am not able to commit to fixing bugs outside of the packaging and I have no intention to change the current workflow that is used in the CI. My patchset would be able to serve both the existing CI and the RPM packaging in openSUSE.

[fsdrw08]

Any updates ?

[SchoolGuy]

My PR was closed without a lot of reasoning. The method I choose still works but the maintenance effort is only possible when upstream accepts my contribution. As such this ticket is impossible to achieve without a change of heart by the Rancher team.

[a1994sc]

Well the existence of RKE2 is nice, having an rpm version of k3s would be absolutely amazing. Well I could implement the work that SchoolGuy did, I would rather have an RPM from the vendor, Rancher.

[github-actions[bot]]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[SchoolGuy]

This is still relevant...

[a1994sc]

Very much still relevant

[c4t3l]

This is still relevant.

[github-actions[bot]]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[a1994sc]

Still an issue

[c4t3l]

This is still a relevant issue, even after two years. I really wish the maintainers would reconsider their position.

[SchoolGuy]

Ping

[cyqsimon]

Bump before the rogue bot gets restless again.

[mildred]

As stated in #630, https://openbuildservice.org/ could be used to build and provide a package for all major distributions.

[SchoolGuy]

This was already mentioned. Upstream Kubernetes uses this build system for over a year now.

[sverkera]

+1 on this topic

[github-actions[bot]]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[c4t3l]

+1 this is still valid

[SchoolGuy]

I am renewing my commitment to the closed PR if upstream decides to accept my contributions.

The proposed solution by my does not interfere with the build system choosen by the k3s maintainers, it merly allows an additonal way that enabled the community to build k3s as an RPM to allow it to integrate with traditional management of system components.

This is beneficial to SUSE as a company as our customers now would be able to use Uyuni/SUMAs Content Lifecycle Management to be able to deploy their SUMA managed edged servers to depend less on custom scripting that is currently used to work around this limitation of k3s, reducing the amout of support for those particular customer(s). Further potential synergies are that k3s could now be integrated with CVE scanners that can crosscheck installed versions of RPMs against the CVE database. Lastly we would benefit from this because we can patch those mentioned CVEs without needing to release a new upstream version of k3s to fix those as the OBS automatically rebuilds RPMs when transitive dependencies change, this should gave package maintainers the peace of mind that they only need to worry about their own source code and not about the entire vendored userspace.

[clintfred]

+1 still relevant. We are looking to use this in space and having a well-maintained deb/rpm package is a must.

[SchoolGuy]

I can again just urge all customers of SUSE/Rancher to contact their corresponding contacts at our company to push this up the chain. I am still internally trying to make this happening but without customers (and the linked revenue) it seems there is no levers I can push at the moment.

I am sorry that this issue is still not implemented but my options from the other side of the business seem limited.

For Rancher folks reading this: I still believe that this is beneficial and I still stand by my offer to rebase my PR anytime you give me the green light. This will not interfere with how you desire to ship your project, it will just offer a second way.

[denisgolius]

For me, It would be easier to create a repository for k3s rpm builds and build rpms via fedora COPR, as I did here.

PS: i'm thinking to do the same for apt repositories

[vincebrannon]

Raised jira SURE-9364