Kubernetes v1.24+ container runtime on Fedora CoreOS

dghubble commented 3 years ago

Kubernetes intends to drop drop support for docker-shim as a container runtime in v1.22. Currently, Fedora CoreOS 33.20210217.3.0 ships docker 19.03.13. docker-shim remains the most stable, tested, available out-of-the-box runtime, but this will end soon. I'd like some kind of clarity on Fedora CoreOS's intentions, such as shipping a compatible containerd or cri-o. With Kubernetes cutting v1.22 alpha releases (likely pretty soon) in the time frame of Fedora 34, as Kubernetes distros we'll want to start evaluating and conformance testing the selection.

Overall, I'd like to know there is some plan. Ideally a documented one. Flatcar Linux has published their intentions to ship containerd in time and already has a mechanism to test it (docs).

cri-o

It sounds like cri-o can only be installed by downloading an RPM (from where?) directly and rpm-ostree installing it (unverified). dnf and yumdownloader used by an openshift script aren't present. https://github.com/coreos/fedora-coreos-tracker/issues/292#issuecomment-796998069. I have some maturity concerns about that. Is this really the recommended path? For a container optimized distro to not have a better path to getting a container runtime?

Releases are also pinned to Kubernetes versions and seem to lag by quite some time, so I have some velocity concerns (the runtime needs to be fairly stable, we roll forward when Kubernetes does, think hours).

containerd

I haven't seen Fedora CoreOS plans to make this the runtime of choice.

Or is the plan something else?

Conan-Kudo commented 3 years ago

In my ideal universe, we would package all the k8s binaries, cri-o and crictl as part of a kubernetes package, all of which would be modular per-release

The kubernetes module should probably have both kubernetes and cri-o. No more separate cri-o module.

dghubble commented 3 years ago

@Conan-Kudo some of us would only want cri-o. Not the rest.

navaati commented 3 years ago

@dghubble As I understand things it's just a module so you can enable it, only install the cri-o package and don't install the kube package and you won't get anything you don't want, I think it's ok ?

Conan-Kudo commented 3 years ago

Yes, indeed. You can effectively treat a module as a subrepo.

haircommander commented 3 years ago

@haircommander, I noticed that f35 only has 1.19, while f34 has 1.20. And f36 only has 1.21. Is this intended or does the module just need some loving?

I don't think I understand how to use modules well haha. It needs some love and some research on my part :)

The kubernetes module should probably have both kubernetes and cri-o. No more separate cri-o module.

@Conan-Kudo I like the sound of that! Do you think it is still useful to have the cri-o module? I can see it making sense to mirror the two if that wasn't too difficult

Conan-Kudo commented 3 years ago

The two ways it could be done:

Kubernetes module includes kubernetes and cri-o, and you can have a separate cri-o module if you'd like for just cri-o
Kubernetes module includes only kubernetes, but depends on separate cri-o module.

The second option makes sense to do, actually, if we expect kubernetes and cri-o updates to happen independently. Having them in one module basically forces that both are built together every time, even if you're only updating one of them.

anthr76 commented 3 years ago

I would be a fan of the second approach personally. Can the cri-o dependency be automatically resolved or will the module need to enabled first?

buckaroogeek commented 3 years ago

Speaking from a cri-o and kubernetes consumer perspective (of the fedora-based home lab variety), I recommend separate modules at this point. CRI-O is still in the incubating stage of adoption (according to cri-o web site) so combining into a unified module might be premature - even though a module can essentially function as a sub repo. Also, cri-o and kubernetes are intended to synchronize on the minor version but patch levels can be different due to the differing release policies of both products.

@haircommander - I would appreciate getting the cri-o module for F34 and F35 caught up to the current release. I know everyone is over committed with tasks however.

thanks

haircommander commented 3 years ago

Kubernetes module includes kubernetes and cri-o, and you can have a separate cri-o module if you'd like for just cri-o

the advantage of this approach is kubernetes package could be modular based on the minor version of kubernetes, and would come with the corresponding cri-o/crictl version. Having them be separate would involve the user correctly installing the correct module.

Alternatively, CRI-O 1.22 module could specify: Provides: kubernetes-cri, kubernetes-cri-1.22 and the kubernetes 1.22 release could rely on kubernetes-cri-1.22 or something

Conan-Kudo commented 3 years ago

The problem is that kubernetes can be used with other CRIs, and we don't currently have our kubernetes package configured to default to CRI-O yet. If we did, then we can do something like Provides: kubernetes(cri) = 1.22 and have the Kubernetes package Requires: (kubernetes(cri) = 1.22 or containerd or moby-engine or docker or docker-ce)

buckaroogeek commented 3 years ago

DNF has a version lock plug-in which can be useful in this type of scenario. The plug-in can be used to block minor level changes but allow patch level changes to a target rpm (or set of rpms).

Conan-Kudo commented 3 years ago

DNF has a version lock plug-in which can be useful in this type of scenario. The plug-in can be used to block minor level changes but allow patch level changes to a target rpm (or set of rpms).

We should not rely on dnf versionlock, especially since rpm-ostree doesn't have that capability.

dustymabe commented 3 years ago

We discussed this in the community meeting today.

13:15:47         dustymabe | #action lorbus and jlebon to reach out to the containers team
                           | to discuss what cri-o versions will be supported and how at the
                           | modular level to pull it off

jlebon commented 2 years ago

Was chatting with @LorbusChris and @haircommander about this. We all think it would be best to keep the kubernetes and cri-o modules separate. @haircommander will keep the cri-o module more up-to-date and match the N-2 support cadence of upstream.

For CI, we're thinking both:

Hook up OKD to f-c-c using Prow. OKD will move to using the cri-o module.
Hook up cri-o integration suite to f-c-c using CoreOS CI or Prow. These are single host tests, so we'd just need to bring up a VM per stream version.

dghubble commented 2 years ago

Checking in about cri-o and its status.

rpm-ostree should support modules (https://github.com/coreos/rpm-ostree/pull/2760)
rpm-ostree module install cri-o:1.20 is an outdated command today (Unknown command 'module' @jlebon)
Upstream tests curl down scripts (non-starter)
Use of modules still seems in-progress (https://github.com/coreos/fedora-coreos-tracker/issues/88, https://github.com/coreos/fedora-coreos-tracker/issues/681)
Test coverage was dropped (https://github.com/coreos/fedora-coreos-config/pull/1211)
Some imperative commands were found:
- sudo rpm-ostree ex module install cri-o:1.20/default (works, but pretty old)
- sudo rpm-ostree ex module install cri-o:1.21/default (fails)
- sudo rpm-ostree ex module install cri-o:1.22/default (fails)
Open bug reports about modules/cri-o not able to be installed

Docs:

FCOS docs don't mention modules (https://github.com/coreos/fedora-coreos-docs)
Modularity docs don't mention FCOS (https://docs.fedoraproject.org/en-US/modularity/)
cri-o docs don't mention FCOS (https://cri-o.io/)
container/cri-o still shows using atomic (https://src.fedoraproject.org/container/cri-o)
modules/cri-o shows no directions (https://src.fedoraproject.org/modules/cri-o/tree/1.22)

The desired end state here is an official Butane config to install a specific version of cri-o reliably.

jlebon commented 2 years ago

Checking in about cri-o and its status.

rpm-ostree should support modules (Add support for modules rpm-ostree#2760)

rpm-ostree module install cri-o:1.20 is an outdated command today (Unknown command 'module' @jlebon)

Sorry about that, my example command in https://github.com/coreos/fedora-coreos-tracker/issues/767#issuecomment-892119669 was missing the ex. I've added that now.

Use of modules still seems in-progress (Modularity for extending the host #88

Thanks, that issue is obsolete IMO. I closed it.

Test coverage was dropped (tests/misc-ro: drop rpm-ostree ex module install test fedora-coreos-config#1211)

It was moved to the rpm-ostree CI: https://github.com/coreos/rpm-ostree/blob/f3a977a32bc76c0776f7d12e5c869d46fda86e9f/tests/kolainst/destructive/layering-modules#L13-L21

Related to this, we're also working on adding k8s e2e tests (which uses cri-o) in https://github.com/coreos/fedora-coreos-config/pull/1297. It doesn't use the Fedora module, but contributes to providing confidence that newer versions won't break.

Some imperative commands were found:

sudo rpm-ostree ex module install cri-o:1.20/default (works, but pretty old)

sudo rpm-ostree ex module install cri-o:1.21/default (fails)

sudo rpm-ostree ex module install cri-o:1.22/default (fails)

Yes, this is an issue (mentioned it higher up too). We chatted recently with @haircommander who's working on updating the module.

FCOS docs don't mention modules (coreos/fedora-coreos-docs)

Yeah, agreed we should add docs there, but sadly I don't think we're quite ready for it yet. I'd like to have updated modules and more tests in CI first.

Modularity docs don't mention FCOS (docs.fedoraproject.org/en-US/modularity)

Modularity is a Fedora effort and not variant-specific, so it wouldn't mention FCOS there.

The desired end state here is an official Butane config to install a specific version of cri-o reliably.

Yes, agreed. :+1:

As you've noticed, there's still a lot of work left here. Thank you for highlighting this!

buckaroogeek commented 2 years ago

I note that sudo rpm-ostree ex module install cri-o:1.21 now works but 1.22 still fails. Fedora 34 only has cri-o 1.21 and older available as modules, whereas Fedora 35 also includes version 1.22.

haircommander commented 2 years ago

I note that sudo rpm-ostree ex module install cri-o:1.21 now works but 1.22 still fails. Fedora 34 only has cri-o 1.21 and older available as modules, whereas Fedora 35 also includes version 1.22.

yeah this is because 1.22 requires go 1.16, which f34 does not seem to have

Conan-Kudo commented 2 years ago

What? Of course Fedora Linux 34 has Go 1.16. We updated to Go 1.16 in F34: https://fedoraproject.org/wiki/Changes/golang1.16

haircommander commented 2 years ago

I stand corrected :slightly_smiling_face: I will look into it

dghubble commented 2 years ago

On F35 too, there are still problems with a manual install of crio 1.22.

NAME="Fedora Linux"
VERSION="35.20211029.3.0 (CoreOS)"

sudo rpm-ostree ex module install cri-o:1.22
NOTICE: Experimental commands are subject to change.
Checking out tree ccdac24... done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2021-09-21T18:07:30Z solvables: 4
rpm-md repo 'fedora-modular' (cached); generated: 2021-10-29T10:04:16Z solvables: 1283
rpm-md repo 'updates-modular' (cached); generated: 2021-12-01T01:40:40Z solvables: 1328
rpm-md repo 'updates' (cached); generated: 2021-12-06T01:00:02Z solvables: 12664
rpm-md repo 'fedora' (cached); generated: 2021-10-29T10:17:40Z solvables: 65732
rpm-md repo 'updates-archive' (cached); generated: 2021-12-06T01:15:12Z solvables: 13006
error: Installing modules: Problems appeared for module install request:
  - Modular dependency problem: Problem during install for module 'cri-o' stream '1.22': No default profile found for cri-o:1.22:3520211111201912:f27b74a8:x86_64

buckaroogeek commented 2 years ago

The steps I took on FCOS f35 for cri-o were:

sudo rpm-ostree ex module enable cri-o:1.22 sudo rpm-ostree install cri-o

On Mon, Dec 6, 2021 at 1:44 PM Dalton Hubble @.***> wrote:

On F35 too, there are still problems with a manual install of crio 1.22.

NAME="Fedora Linux" VERSION="35.20211029.3.0 (CoreOS)"

sudo rpm-ostree ex module install cri-o:1.22 NOTICE: Experimental commands are subject to change. Checking out tree ccdac24... done Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora updates-archive Importing rpm-md... done rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2021-09-21T18:07:30Z solvables: 4 rpm-md repo 'fedora-modular' (cached); generated: 2021-10-29T10:04:16Z solvables: 1283 rpm-md repo 'updates-modular' (cached); generated: 2021-12-01T01:40:40Z solvables: 1328 rpm-md repo 'updates' (cached); generated: 2021-12-06T01:00:02Z solvables: 12664 rpm-md repo 'fedora' (cached); generated: 2021-10-29T10:17:40Z solvables: 65732 rpm-md repo 'updates-archive' (cached); generated: 2021-12-06T01:15:12Z solvables: 13006 error: Installing modules: Problems appeared for module install request:

Modular dependency problem: Problem during install for module 'cri-o' stream '1.22': No default profile found for cri-o:1.22:3520211111201912:f27b74a8:x86_64

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/coreos/fedora-coreos-tracker/issues/767#issuecomment-987260819, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKDYHXLSHEVQ4LMA4M4RSLUPUVDVANCNFSM4ZBN6ONA .

dustymabe commented 2 years ago

Hey @dghubble - we identified this problem last week: in the cri-o module there is a profile called default but it's not explicitly marked as the default so you end up with the problem you observed. We're working with @haircommander to get the default profile marked as default.

For now you can do something like what @buckaroogeek mentioned above or in one command:

[core@cosa-devsh ~]$ sudo rpm-ostree ex module install cri-o:1.22/default
NOTICE: Experimental commands are subject to change.
Checking out tree 13af377... done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2021-09-21T18:07:30Z solvables: 4
rpm-md repo 'fedora-modular' (cached); generated: 2021-10-29T10:04:16Z solvables: 1283
rpm-md repo 'updates-modular' (cached); generated: 2021-12-01T01:40:40Z solvables: 1328
rpm-md repo 'updates' (cached); generated: 2021-12-07T01:51:30Z solvables: 12820
rpm-md repo 'fedora' (cached); generated: 2021-10-29T10:17:40Z solvables: 65732
rpm-md repo 'updates-archive' (cached); generated: 2021-12-07T02:18:41Z solvables: 13149
Resolving dependencies... done
Will download: 2 packages (34.6?MB)
Downloading from 'updates-modular'... done
Importing packages... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Added:
  cri-o-1.22.1-1.module_f35+13370+fa0e2e9f.x86_64
  cri-tools-1.22.0-3.module_f35+13370+fa0e2e9f.x86_64

jlebon commented 2 years ago

For now you can do something like what @buckaroogeek mentioned above or in one command:

Note that the semantics are not the same between these two approaches. rpm-ostree ex module install installs profiles, which is a set of packages not necessarily dependent on each other. On the other hand, rpm-ostree ex module enable only enables a stream and rpm-ostree install may install any packages from that stream; only its dependencies will be pulled in.

As a concrete example, the steps from @buckaroogeek will not pull in cri-tools since it's not a strict dependency of cri-o, while those from @dustymabe will since it's part of the default profile.

Richterrettich commented 2 years ago

Does this also imply that in the future, the preferred method of installation for Kubernetes is module based too? Otherwise, it would not easily be possible to keep crio/kubernetes versions in sync.

LorbusChris commented 2 years ago

I agree that a Kubernetes module would be the sanest approach to deal with multiple versions here. I think @janch32 is the maintainer of the RPM in Fedora today. It's probably best to work with him to get that RPM modularized.

LorbusChris commented 2 years ago

Sorry, I tagged the Jan (my apologies!). The one I meant should be @ingvagabund

buckaroogeek commented 2 years ago

There is an existing, but orphaned, module package for Kubernetes in the fedora package management system. I ran across it while exploring k8 installation on CoreOS using fedora rpms vs upstream rpms. Given the current support policy from kubernetes (the last 3 minor releases - so 1.21, 1.22, 1.23 today for example) and the number of patch releases in each minor release, a modular approach would be very useful.

dghubble commented 2 years ago

Followup on https://github.com/coreos/fedora-coreos-tracker/issues/767#issuecomment-967793379

The desired end state here is an official Butane config to install a specific version of cri-o reliably

We still lack a declarative butane config for getting cri-o. Kubernetes v1.24 will remove dockershim.

Manual commands (which we don't ultimately want to resort to) don't seem ready (Kubernetes v1.23 came out 31 days ago).

sudo rpm-ostree ex module enable cri-o:1.23                               
NOTICE: Experimental commands are subject to change.
Checking out tree 30c82ee... done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2021-09-21T18:07:30Z solvables: 4
rpm-md repo 'fedora-modular' (cached); generated: 2021-10-26T05:08:36Z solvables: 1283
rpm-md repo 'updates-modular' (cached); generated: 2022-01-06T01:57:22Z solvables: 1371
rpm-md repo 'updates' (cached); generated: 2022-01-08T01:04:36Z solvables: 16018
rpm-md repo 'fedora' (cached); generated: 2021-10-26T05:31:27Z solvables: 65732
rpm-md repo 'updates-archive' (cached); generated: 2022-01-08T01:31:50Z solvables: 18937
error: Problems appeared for module enable request:
  - Unable to resolve argument 'cri-o:1.23'

less /etc/os-release 
NAME="Fedora Linux"
VERSION="35.20211215.3.0 (CoreOS)"

dustymabe commented 2 years ago

hmm. There's activity on the 1.23 branch, but I don't see any releases of the module. @haircommander - can you put out a release?

LorbusChris commented 2 years ago

Go 1.17 won't land in Fedora 35, so there's likely not going to be an official release of cri-o 1.23 until F36 :(

Conan-Kudo commented 2 years ago

Well, modular Go can be used to build cri-o:1.23 module on F35, couldn't it?

LorbusChris commented 2 years ago

It could, yes, but right now there's no Golang module being maintained for Fedora

ingvagabund commented 2 years ago

Hello everyone

I agree that a Kubernetes module would be the sanest approach to deal with multiple versions here. I think @janch32 is the maintainer of the RPM in Fedora today. It's probably best to work with him to get that RPM modularized.

Unfortunately, I am no longer active in the RPM Fedora area. The Kubernetes rpms in Fedora has been lacking maintenance for some time. I recall there was an effort to make Kubernetes modular. However, I was not part of the effort. If anyone is interested in proceeding here and keeping the Kubernetes rpms up2date, I can share the access rights.

travier commented 2 years ago

Commands for 1.22: https://github.com/coreos/fedora-coreos-config/pull/1406

anthr76 commented 2 years ago

Hello everyone

I agree that a Kubernetes module would be the sanest approach to deal with multiple versions here. I think @janch32 is the maintainer of the RPM in Fedora today. It's probably best to work with him to get that RPM modularized.

Unfortunately, I am no longer active in the RPM Fedora area. The Kubernetes rpms in Fedora has been lacking maintenance for some time. I recall there was an effort to make Kubernetes modular. However, I was not part of the effort. If anyone is interested in proceeding here and keeping the Kubernetes rpms up2date, I can share the access rights.

I have not worked directley with Fedora RPMs but familiar of the process this is something I would be interested in. I'm on matrix if you wanted to speak directley @anthr76:mozilla.org

buckaroogeek commented 2 years ago

I have the time, interest, and (some) skills to also contribute where I can add value.

Brad

haircommander commented 2 years ago

So it sounds like our eventual ideal situation is having modular go, cri-o and kubernetes. We have modular cri-o, so ideally someone could add and maintain go and kubernetes. Do either of those tasks interest either of you @buckaroogeek @anthr76 ?

buckaroogeek commented 2 years ago

Sure I, for one, can give it a go.

:)

Actually, yes I am very interested in helping. Reviewing the package maintainer web pages now. And will follow through on the needed steps. I might also suggest helm as that seems to also be lagging in packaging support. It would seem that getting some solid traction around go and a go module would be a first step? After getting the package maintainers introductions done.

dustymabe commented 2 years ago

@haircommander So it sounds like our eventual ideal situation is having modular go, cri-o and kubernetes.

It looks like the golang maintainers started a discussion about golang and different versions recently on Fedora Devel. It looks like the consensus there was to use golang compat packages (not a module) if they wanted to support different versions in Fedora. Which IIUC should be fine for what we need.

I think the status is:

golang
- work with maintainers to help them achieve compat packages
  - email asm@redhat.com to get involved
kubernetes
- create a kubernetes module.
  - work with @haircommander (pehunt@redhat.com) and @ingvagabund (jchaloup@redhat.com)
crio
- @haircommander should have this one covered.

@buckaroogeek @anthr76 - does that sound good?

buckaroogeek commented 2 years ago

@dustymabe - Yes, it does for me.

dghubble commented 2 years ago

These efforts sound like the right direction to getting cri-o to be a better option on Fedora CoreOS. However, I'm planning to migrate Typhoon users to containerd on Fedora CoreOS. I don't want to distract from this issue's focus (getting one or more first class container runtimes on FCOS), but for those interested, I outlined why here.

Conan-Kudo commented 2 years ago

I can help with the non-modular kubernetes and cri-o packages that are required to be present alongside modular ones.

ingvagabund commented 2 years ago

@buckaroogeek @anthr76 feel free to share you fedora usernames so I can add you as committers to the kubernetes project. Checking the previous commits might give you some hints on how to update the kubernetes to the next version. I have not installed the kubernetes rpms for ages so it might be challenging at the beginning to make it work. Though learning from mistakes will eventually yield the right results.

buckaroogeek commented 2 years ago

@ingvagabund - buckaroogeek is the name I use in FAS.

Thanks for the tips and assist.

ingvagabund commented 2 years ago

When adding you as a committer, I am getting This user must be in one of the following groups to be allowed to be added to this project: packager. I believe you need a sponsor. Not sure what's the process these days to obtain one.

buckaroogeek commented 2 years ago

@ingvagabund - I do not want to further side track this issue. May I contact you directly? A ticket needs to be filed with the package sponsor pagure instance by a current sponsor.

anthr76 commented 2 years ago

@buckaroogeek @anthr76 feel free to share you fedora usernames so I can add you as committers to the kubernetes project.

My fas is anthr76

I can be reached on matrix as @anthr76:mozilla.org

Thanks!

ingvagabund commented 2 years ago

I do not want to further side track this issue. May I contact you directly?

@buckaroogeek @anthr76 what channel do you prefer? IRC, email, ...?

anthr76 commented 2 years ago

I do not want to further side track this issue. May I contact you directly?

@buckaroogeek @anthr76 what channel do you prefer? IRC, email, ...?

I'm on matrix as @anthr76:mozilla.org I'm in #fedora-coreos and #fedora-golang which is also bridged with IRC

coreos / fedora-coreos-tracker

Kubernetes v1.24+ container runtime on Fedora CoreOS #767

cri-o

containerd