search

cloud-ark / kubeplus

Kubernetes Operator for multi-instance multi-tenancy
https://cloudark.io/
Apache License 2.0
660 stars 79 forks source link

Operator projects using the removed APIs in k8s 1.22 requires changes. #930

Closed camilamacedo86 closed 1 year ago

camilamacedo86 commented 3 years ago

Problem Description

Kubernetes has been deprecating API(s), which will be removed and are no longer available in 1.22. Operators projects using these APIs versions will not work on Kubernetes 1.22 or any cluster vendor using this Kubernetes version(1.22), such as OpenShift 4.9+. Following the APIs that are most likely your projects to be affected by:

Therefore, looks like this project distributes solutions via the Red Hat Connect with the package name as kubeplus and does not contain any version compatible with k8s 1.22/OCP 4.9. Following some findings by checking the distributions published:

NOTE: The above findings are only about the manifests shipped inside of the distribution. It is not checking the codebase.

How to solve

It would be very nice to see new distributions of this project that are no longer using these APIs and so they can work on Kubernetes 1.22 and newer and published in the Red Hat Connect collection. OpenShift 4.9, for example, will not ship operators anymore that do still use v1beta1 extension APIs.

Due to the number of options available to build Operators, it is hard to provide direct guidance on updating your operator to support Kubernetes 1.22. Recent versions of the OperatorSDK greater than 1.0.0 and Kubebuilder greater than 3.0.0 scaffold your project with the latest versions of these APIs (all that is generated by tools only). See the guides to upgrade your projects with OperatorSDK Golang, Ansible, Helm or the Kubebuilder one. For APIs other than the ones mentioned above, you will have to check your code for usage of removed API versions and upgrade to newer APIs. The details of this depend on your codebase.

If this projects only need to migrate the API for CRDs and it was built with OperatorSDK versions lower than 1.0.0 then, you maybe able to solve it with an OperatorSDK version >= v0.18.x < 1.0.0:

$ operator-sdk generate crds --crd-version=v1 INFO[0000] Running CRD generator.
INFO[0000] CRD generation complete.

Alternatively, you can try to upgrade your manifests with controller-gen (version >= v0.4.1) :

If this project does not use Webhooks:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role paths="./..."

If this project is using Webhooks:

  1. Add the markers sideEffects and admissionReviewVersions to your webhook (Example with sideEffects=None and admissionReviewVersions={v1,v1beta1}: memcached-operator/api/v1alpha1/memcached_webhook.go):

  2. Run the command:

    $ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role webhook paths="./..."

For further info and tips see the blog.

Thank you for your attention.

devdattakulkarni commented 3 years ago

@camilamacedo86 Thank you for the heads up. Can you let me know the timeline for Openshift 4.9 release and when the migration of the deprecated APIs need to be completed?

camilamacedo86 commented 3 years ago

Hi @devdattakulkarni,

That is described in the blog.

Uploading a new version of your Operator by September 30th is key to supporting a smooth upgrade path for your customers and preventing confusion. If you do not push new versions compatible with 4.9 by September 30th, your package will be completely removed from OpenShift 4.9 until you have an updated version.

Suppose you do not push new versions compatible with OCP 4.9 in time. Then, you will still be able to do that, however, you will need to be aware of some specifics configurations to ensure that you have a compatible version targeting 4.8 (at least) and 4.9+. In this way, the best way would publish a new version before that.

Do you have any questions about how to do that? Do you think that the steps added to this issue can help you out? Has anything else that I can do to help you out with?

devdattakulkarni commented 3 years ago

@camilamacedo86 Thanks for the quick reply. I think the pointers that you have provided are good starting point. We will go through them and if run into any difficulties then follow up here. One question that I do have is - in case we are not able to complete the migration and certification before September 30th, would KubePlus SaaS Manager get removed from Red Hat Marketplace completely or it will remain on RHCP but just not be shown to OCP 4.9 users and will still be available to users who are using prior versions of OCP.

camilamacedo86 commented 3 years ago

Hi @devdattakulkarni,

That means remove your package only from OCP 4.9+ catalogs. Nothing will change in the previous OCP versions where they work.

devdattakulkarni commented 3 years ago

@camilamacedo86 Got it. That is good to know.

camilamacedo86 commented 3 years ago

Hi, would we have any update on this?

See that we are very close to the release data and fix the projects seems not very hard. See how to fix it in the first comment. Then, would be great to be able to check a new version of your project distributed which is compatible with 4.9.

devdattakulkarni commented 3 years ago

@camilamacedo86 Thanks for checking in.

While I know that the change is not big, there is some code change needed on our end since we refer to the CRD APIs in our code via the apiextensions v1beta1 paths. All such places in the code need to be updated as well. But if we do that I am not sure if the Operator code would work with the prior versions of Kubernetes or Openshift (4.7, 4.8) etc.? Or do we need to have two separate versions one for OpenShift < 4.9 and one for OpenShift >= 4.9? Do you have any thoughts/suggestions how to make the changes such that we don't need to maintain two separate copies of our code files and at the same time support Kubernetes versions prior to 1.22 (and OpenShift versions 4.7, 4.8 and 4.9)?

camilamacedo86 commented 3 years ago

Hi @devdattakulkarni,

. But if we do that I am not sure if the Operator code would work with the prior versions of Kubernetes or Openshift (4.7, 4.8) etc.? Or do we need to have two separate versions one for OpenShift < 4.9 and one for OpenShift >= 4.9?

The latest versions of the k8s api (v1) work well in all OCP supported versions. You do not need to have 2 bundles at all. The same bundle will work fine for all of them.

Do you have any thoughts/suggestions how to make the changes such that we don't need to maintain two separate copies of our code files and at the same time support Kubernetes versions prior to 1.22 (and OpenShift versions 4.7, 4.8 and 4.9)?

Just upgrade your project, generate a new bundle and publish asap. Note that we will need ultimate deprecate from 4.9 all versions that we are able to programmatically identify that does not work on 4.9 at all which is your case. Thus, I would recommend you push the new version asap because if not, your package/project will get removed from 4.9 catalog and then you will be unable to use the replaces config only your first compatible version shipped on 4.9. That is the easy path. Publish asap a new version compatible with 4.9 and configured to be on OCP versions or at least for 4.8.

devdattakulkarni commented 3 years ago

Just upgrade your project, generate a new bundle and publish asap. Note that we will need ultimate deprecate from 4.9 all versions that we are able to programmatically identify that does not work on 4.9 at all which is your case. Thus, I would recommend you push the new version asap because if not, your package/project will get removed from 4.9 catalog and then you will be unable to use the replaces config only your first compatible version shipped on 4.9. That is the easy path. Publish asap a new version compatible with 4.9 and configured to be on OCP versions or at least for 4.8.

We will upgrade the version in coming days. That said, I did not understand the part about replaces config. Can you elaborate on what do you mean by that?

camilamacedo86 commented 3 years ago

Hi @devdattakulkarni.

If you do not push that ASAP ( see that the ETA for you achieve this goal already passed ) then, be aware that your first publication targeting OCP 4.9 cannot use the “replaces” configuration since any previous version published so far does not exist on 4.9 to be replaced. You can publish your new distribution using the skips or skipRange options instead of replaces. More info: https://v0-18-z.olm.operatorframework.io/docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph/

All please see https://connect.redhat.com/blog/api-deprecation-kubernetes-122-will-impact-your-operators.

enyachoke commented 2 years ago

Hi @devdattakulkarni will you be upgrading the project to support 1.22?

devdattakulkarni commented 2 years ago

Hi @enyachoke Yes! We will be doing that soon. In what context are you using KubePlus? If you are currently able to use the 1.21 version of Kubernetes, I would suggest using that. Let us know.

enyachoke commented 2 years ago

Thanks @devdattakulkarni I just found the project last week and I wanted to spin up a cluster to evaluate it I think I used 1.23. I can definitely spin one with 1.21. Super excited about what I could be able to achieve with the project

devdattakulkarni commented 2 years ago

Hi @camilamacedo86 We have now migrated KubePlus to be compatible with Kubernetes 1.24. I was wondering if you can provide pointers regarding how to get it re-certified with OpenShift and get it ready for RedHat Marketplace.

Regards.