Installation docs: What version of Kubernetes is required / recommended?

[roberthbailey]

From https://development.open-match.dev/site/docs/installation/:

There are instructions on how to create a GKE cluster (and links to how to create minikube) but no indication of what version of Kubernetes should be used.

My suggestion (this is what we do for Agones) would be to pick a version and just suggest using that one. Use it in your CI and developer testing and that will give you confidence that it works. It also allows you to tailor your instructions based on a specific version of Kubernetes (e.g. https://github.com/googleforgames/open-match-docs/issues/84).

[Laremere]

If someone wants to use both Agones and Open Match in the same cluster, they might force incompatible k8s versions for the latest Agones and OM versions.

[roberthbailey]

That's true. Some options:

1. Run OM and Agones in different clusters
2. Pick a version and qualify them in that version yourself (presumably you pick a version supported by one or the other to reduce your burden)
3. Ask OM + Agones devs to keep the versions in sync.

[Laremere]

Warning: bit of train of thought and research here:

I think for OM, it has a lot less surface area for interacting with k8s. So it's much more reasonable to support a range of versions.

k8s lists it supporting 3 versions at a time: (currently 1.16, 1.15, 1.14?) https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release/versioning.md#kubernetes-release-versioning

Though gke is lagging behind, currently allowing selection between: 1.12, 1.13, and 1.14 EKS is currently: 1.11, 1.12, 1.13, and 1.14 AKS seems to be stating it currently supports: 1.10, 1.11, 1.12, 1.13, and 1.14; however it wants me to run a command based on region to get the correct answer.

So the k8s ideal doesn't appear to be what used in practice.

As far as deprecation goes, the recent 1.16 release had a few: https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

The most recent replacement was available at the earliest in k8s v1.10. So the deprecation policy seems to be generous enough that supporting the newest version will only eliminate versions not supported by any cloud provider anyways.

The Agones documentation states:

Later versions of Kubernetes may work, but this project is tested against 1.12, and is therefore the supported version. Agones will update its support to n-1 version of what is available across all major cloud providers - GKE, EKS and AKS

That seems like a reasonable version choice. If we have a range of supported versions, that forms a good lowest supported version number. Though there's still a problem where OM updates before Agones, and you can't use the new OM version until Agones catches up.

So maybe we should state what version of k8s we test against. However we've gotten a bug in the past about things not working with the latest minikube version. So I think it's a good idea to support the latest k8s version too. In a conflict, the practical choice is to continue supporting the version that actually runs on cloud providers.

As for supporting the latest k8s version in practice, I'm not entirely sure. We would need some way to test multiple new versions, including those not yet on GKE (which we currently use for testing.)
I think it would be good to have a more thorough version testing ability. Probably using minikube (since that seems to be very up to date). That way we can test against a skew of versions. This really only needs to be done before a release, as the parts of k8s we use are stable enough that they really shouldn't be changing behavior very often.

So my current recommendation:

So, I think for v0.9, we align with Agones' supported version statement (not necessarily the version supported by the latest Agones release) in the testing, and we put that note in the documentation.

Then at some point in the future, we should get version skew testing, and expand the officially supported versions from the latest k8s version (at time of OM's minor, not patch, release) all the way to Agones' supported version statement. This should probably be done before 1.0, as it has an effect on being able to practically run this in production.

[roberthbailey]

You have some great points. I agree that since OM is much less tied in with k8s internals, it's much less dependent on the version of k8s that is running in the cluster. However, I think it's more about how large of a test matrix you want to support rather than how many you think will work. It comes down to whether you've tested the matrix (and can keep testing it in CI).

However we've gotten a bug in the past about things not working with the latest minikube version.

Since your directions for using minikube imply to just install the latest, users would expect it to work with the latest version. If you specify which k8s version to use in minikube (via a command line flag) then you can tell people how to install & test with minikube and ensure that they are testing against a version that you support.

We would need some way to test multiple new versions, including those not yet on GKE (which we currently use for testing.)

KIND might be a good way to do this, since it's being used for more and more of the k8s testing (at head) so it supports much more recent versions, and it's easy to use in CI (especially prow).

Overall, +1 to your recommendation. Getting skew testing would be great (maybe a goal prior to 1.0?) but I don't think it should block the next release. Making things clear to early adopters or people doing testing to kick the tires is more important than having a wide variety of versions that you support today. And if you show on your roadmap that you are working towards supporting more versions that's even better.

[yfei1]

For now I'll add a prerequisite section with the tested GKE+Minikube versions listed. Will discuss this with the team offline in 0.9.