search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.5k stars 4.89k forks source link

Kicbase: update cri-o packaging sources #19504

Open JanGutter opened 2 months ago

JanGutter commented 2 months ago
k8s-ci-robot commented 2 months ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: JanGutter Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/kubernetes/minikube/blob/master/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
k8s-ci-robot commented 2 months ago

Welcome @JanGutter!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. :smiley:

k8s-ci-robot commented 2 months ago

Hi @JanGutter. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
minikube-bot commented 2 months ago

Can one of the admins verify this patch?

JanGutter commented 2 months ago

I'd be happy to chat on slack, or open an issue, if needed.

afbjorklund commented 2 months ago

Note that the new packages are only available for "new" versions of Kubernetes.

https://kubernetes.io/blog/2023/10/10/cri-o-community-package-infrastructure/

So versions 1.24-1.27 still use the old repo, versions 1.28-1.31 use the new repo(s).

New packages contain the same binaries as the github releases / GCS cri-o/artifacts.

Replacing the other packages such as podman will take more effort, and to fix the bugs...

trying to overwrite '/usr/libexec/crio/conmon', which is also in package conmon

It would also be good if podman could be updated from 3.4.x to 4.9.x, even it not required.

afbjorklund commented 2 months ago

I'd be happy to chat on slack, or open an issue, if needed.

Podman needs a way out of "experimental" in the roadmap.

This sort-of involves also the CRI-O container runtime, at least as long as they share image storage (Similar to how docker and containerd are connected by optionally using the same image storage...)

So that is something for https://minikube.sigs.k8s.io/community/

There is also some discussion in the upstream Kubernetes project SIGs, if it is "included" in the k8s release. The moving of the packages and the release script is one step into make it more standard, but it missed 1.31.

afbjorklund commented 2 months ago
afbjorklund commented 2 months ago

The new packaging format depends on some k8s distro packages.

That sounds like a bug, the cri-o package should not have any hard dependencies towards kubernetes packages. It does depend on CRI and CNI during runtime, but those tools do not have to be provided by the other repository.

Depends: systemd-sysv,systemd,iptables
Recommends: kubernetes-cni

This makes the assumption that the most recent stable k8s is expected to work with the most recent stable cri-o.

That is not currently the case, the stable k8s was tested with a prerelease of cri-o. Users are supposed to use stable. This means that they will (for some weeks after the release) need to run the new k8s with the old (stable) cri-o.

KUBERNETES_VERSION=v1.31
CRIO_VERSION=v1.30
JanGutter commented 2 months ago

The new packaging format depends on some k8s distro packages.

That sounds like a bug, the cri-o package should not have any hard dependencies towards kubernetes packages. It does depend on CRI and CNI during runtime, but those tools do not have to be provided by the other repository.

Yeah, that's probably due to my misinterpretation of the instructions at https://github.com/cri-o/packaging

This makes the assumption that the most recent stable k8s is expected to work with the most recent stable cri-o.

That is not currently the case, the stable k8s was tested with a prerelease of cri-o. Users are supposed to use stable. This means that they will (for some weeks after the release) need to run the new k8s with the old (stable) cri-o.

My assumption stemmed from the same source: it looks like they list the latest stable series of k8s (1.31) plus the latest stable series of cri-o (1.30) in the example, leading to my assumption that this is expected to work.

JanGutter commented 2 months ago

Note that the new packages are only available for "new" versions of Kubernetes.

https://kubernetes.io/blog/2023/10/10/cri-o-community-package-infrastructure/

So versions 1.24-1.27 still use the old repo, versions 1.28-1.31 use the new repo(s).

The old repos went through a bit of instability a month or so ago - incidentally, this was one of the reasons I started going down this rabbit hole.

afbjorklund commented 2 months ago

The old repos went through a bit of instability a month or so ago

There has been a lot of stability issues with OBS packages over the years.

I think that the new kicbase packaging is unlikely to use the pkgs.k8s.io.

Probably go for something similar to dl.k8s.io instead, also for cri-o artifacts.

BIN_URL="https://storage.googleapis.com/cri-o/artifacts/$COMMIT/$ARCH"

See https://github.com/cri-o/cri-o/releases for the available versions in GCS.

afbjorklund commented 2 months ago

Added this PR for discussion in the community meeting and the office hours, if cri-o should be prioritized...

Last time, we added improved support for Podman to the minikube roadmap - maybe also add CRI-O too ?

2021 datadog: https://www.datadoghq.com/container-report-2021/

2023 datadog: https://www.datadoghq.com/container-report/

You can clearly see cri-o being utilized more, right ? Like the Linux desktop...

https://linux.slashdot.org/story/24/08/01/2029219/linux-hits-another-desktop-market-share-record

JanGutter commented 2 months ago

Oh man, slashdot is still going!

This whole rabbithole started when I tried to get a minimal cluster running with centos 9, podman and an ingress. The nice feature of cri-o was that I could configure registry mirrors to rewrite both docker.io and quay.io fqdn paths, so with a CI job and no manifest rewrites, I could do a test of a kubernetes operator and a number of containers.

JanGutter commented 2 months ago

(This open issue put me on the path to podman https://github.com/moby/moby/issues/18818 ), I'll have to check if containerd's rewrite options are similar. Honestly, if containerd for rootful and rootless podman works better, I'm much less interested in getting cri-o to work.

k8s-ci-robot commented 2 months ago

PR needs rebase.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.