RbacAuthorizationV1alpha1 deprecated from kubernetes v1.22 onwards - pytest fails with AttributeError

[MuraliK7hpe]

kubetest fails to run in latest version on K8s and following is the error observed

File "/usr/local/lib/python3.6/site-packages/kubetest/objects/clusterrolebinding.py", line 12, in class ClusterRoleBinding(ApiObject): File "/usr/local/lib/python3.6/site-packages/kubetest/objects/clusterrolebinding.py", line 30, in ClusterRoleBinding 'rbac.authorization.k8s.io/v1alpha1': client.RbacAuthorizationV1alpha1Api, AttributeError: module 'kubernetes.client' has no attribute 'RbacAuthorizationV1alpha1Api'

Refer to K8s page: https://kubernetes.io/docs/reference/using-api/deprecation-guide/

RBAC resources The rbac.authorization.k8s.io/v1beta1 API version of ClusterRole, ClusterRoleBinding, Role, and RoleBinding is no longer served as of v1.22.

Migrate manifests and API clients to use the rbac.authorization.k8s.io/v1 API version, available since v1.8. All existing persisted objects are accessible via the new APIs No notable changes

[tyxeron]

There are many other deprecations and issues. I tried to monkey patch the issue described above and got other errors of the same kind

[timblaktu]

I hit this error, pointing my tests' kubeconfig/context at a v1.21 k8s cluster.

TL;DR: How do I find the version of kubetest (and accompanying kubernetes client package) that officially supports kubernetes v1.21? Neither 0.9.5 nor 0.9.4 seem to fit the bill.

I have enabled debug logging, but do not see any output until the exception is hit (expand excerpt below for detail).

I can talk to this cluster just fine using kubectl. Since my cluster is using v1.21, perhaps my setup is trying to use a newer Api version? Is a way to specify the Api version that kubetest's client should use? I haven't found anything in the kubetest docs about this, and assume that it automatically detects the k8s version.

Click to expand pytest command and error

```bash $ poetry run pytest --kube-log-level=debug eks/test Traceback (most recent call last): File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/bin/pytest", line 8, in sys.exit(console_main()) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 187, in console_main code = main() File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 145, in main config = _prepareconfig(args, plugins) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 324, in _prepareconfig config = pluginmanager.hook.pytest_cmdline_parse( File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 55, in _multicall gen.send(outcome) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/helpconfig.py", line 102, in pytest_cmdline_parse config: Config = outcome.get_result() File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result raise ex[1].with_traceback(ex[2]) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1016, in pytest_cmdline_parse self.parse(args) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1304, in parse self._preparse(args, addopts=addopts) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1187, in _preparse self.pluginmanager.load_setuptools_entrypoints("pytest11") File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints plugin = ep.load() File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load module = import_module(match.group('module')) File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1030, in _gcd_import File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/plugin.py", line 17, in from kubetest import errors, markers File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/markers.py", line 9, in from kubetest import manager File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/manager.py", line 8, in from kubetest import client, objects, utils File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/client.py", line 13, in from kubetest import objects, utils File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/__init__.py", line 6, in from .clusterrolebinding import ClusterRoleBinding File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/clusterrolebinding.py", line 12, in class ClusterRoleBinding(ApiObject): File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/clusterrolebinding.py", line 30, in ClusterRoleBinding "rbac.authorization.k8s.io/v1alpha1": client.RbacAuthorizationV1alpha1Api, AttributeError: module 'kubernetes.client' has no attribute 'RbacAuthorizationV1alpha1Api' ```

My test itself is trivial. Since it's failing before ever loading my test, I believe my test code is irrelevant.

I'm using python 3.9 and kubetest 0.9.5 on Debian 11 and the following python package versions installed via poetry:

Click to expand details about my environment

```bash $ poetry show attrs 21.4.0 Classes Without Boilerplate boto3 1.21.42 The AWS SDK for Python botocore 1.24.42 Low-level, data-driven core of boto 3. cachetools 5.0.0 Extensible memoizing collections and decorators certifi 2021.10.8 Python package for providing Mozilla's CA Bundle. cfgv 3.3.1 Validate configuration and produce human readable error messages. charset-normalizer 2.0.12 The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet. distlib 0.3.4 Distribution utilities filelock 3.6.0 A platform independent file lock. google-auth 2.6.5 Google Authentication Library identify 2.4.12 File identification library for Python idna 3.3 Internationalized Domain Names in Applications (IDNA) iniconfig 1.1.1 iniconfig: brain-dead simple config-ini parsing jmespath 1.0.0 JSON Matching Expressions kubernetes 23.3.0 Kubernetes python client kubetest 0.9.5 A Kubernetes integration test framework in Python. nodeenv 1.6.0 Node.js virtual environment builder oauthlib 3.2.0 A generic, spec-compliant, thorough implementation of the OAuth request-signing logic packaging 21.3 Core utilities for Python packages platformdirs 2.5.2 A small Python module for determining appropriate platform-specific dirs, e.g. a "user data dir". pluggy 1.0.0 plugin and hook calling mechanisms for python pre-commit 2.18.1 A framework for managing and maintaining multi-language pre-commit hooks. py 1.11.0 library with cross-python path, ini-parsing, io, code, log facilities pyasn1 0.4.8 ASN.1 types and codecs pyasn1-modules 0.2.8 A collection of ASN.1-based protocols modules. pyparsing 3.0.8 pyparsing module - Classes and methods to define and execute parsing grammars pytest 7.1.1 pytest: simple powerful testing with Python python-dateutil 2.8.2 Extensions to the standard Python datetime module pyyaml 6.0 YAML parser and emitter for Python requests 2.27.1 Python HTTP for Humans. requests-oauthlib 1.3.1 OAuthlib authentication support for Requests. rsa 4.8 Pure-Python RSA implementation s3transfer 0.5.2 An Amazon S3 Transfer Manager six 1.16.0 Python 2 and 3 compatibility utilities toml 0.10.2 Python Library for Tom's Obvious, Minimal Language tomli 2.0.1 A lil' TOML parser urllib3 1.26.9 HTTP library with thread-safe connection pooling, file post, and more. virtualenv 20.14.1 Virtual Python Environment builder websocket-client 1.3.2 WebSocket client for Python with low level API options ```

[timblaktu]

@MuraliK7hpe @tyxeron, it appears that the kubetest client object wrapping clusterrolebinding has a list of api versions, with specification for a "preferred" one:

class ClusterRoleBinding(ApiObject):
<snip>
    api_clients = {
        "preferred": client.RbacAuthorizationV1Api,
        "rbac.authorization.k8s.io/v1": client.RbacAuthorizationV1Api,
        "rbac.authorization.k8s.io/v1alpha1": client.RbacAuthorizationV1alpha1Api,
        "rbac.authorization.k8s.io/v1beta1": client.RbacAuthorizationV1beta1Api,
    }

Could there be a problem with this "preferred" mechanism? Why would we be getting this error if the client chose the preferred version first, i.e. rbac.authorization.k8s.io/v1?

EDIT: I found this description in the ApiObject.api_client comment:

A mapping of all the supported api clients for the API object type. Various resources can have multiple versions, e.g. "apps/v1", "apps/v1beta1", etc. The preferred version for each resource type should be defined under the "preferred" key. The preferred API client will be used when the apiVersion is not specified for the resource.

This all appears to be the case for all the api objects in kubetest. I suspect we're just hitting clusterrolebinding first bc it's the first to be imported in the objects package.

[timblaktu]

For a second, I thought that these client api objects were choosing v1alpha1 because I have that specified in my kubeconfig file. I'm using EKS, and the aws eks update-kubeconfig command to generate the kubeconfig for me. In the generated file, it specifies to use v1alpha1 in the command to fetch the auth token for the generated user:

Click to expand user kubeconfig snippet generated by aws eks update-kubeconfig

```yaml users: - name: arn:aws:eks:::cluster/ user: exec: apiVersion: client.authentication.k8s.io/v1alpha1 args: - --region - us-east-1 - eks - get-token - --cluster-name - command: aws env: - name: AWS_PROFILE value: ***** interactiveMode: IfAvailable provideClusterInfo: false ```

However, I'm still getting the same kubetest exception after changing the user-exec apiVersion in my kubeconfig file to v1. :-(

In fact, changing this api version in my kubeconfig also breaks kubectl. It appears that this api version is the only one supported in my cluster for the initial fetch of api versions that kubectl does:

error: couldn't get available api versions from server: Get "https://blahblahblah.eks.amazonaws.com/api?timeout=32s": getting credentials: exec plugin is configured to use API version client.authentication.k8s.io/v1beta1, plugin returned version client.authentication.k8s.io/v1alpha1

My cluster is v1.21, which supports the following api versions per-resource:

Click to expand kubectl api-versions

```bash ~/src/infra2 530-implemen > kubectl api-versions admissionregistration.k8s.io/v1 admissionregistration.k8s.io/v1beta1 apiextensions.k8s.io/v1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 autoscaling/v2beta2 batch/v1 batch/v1beta1 certificates.k8s.io/v1 certificates.k8s.io/v1beta1 coordination.k8s.io/v1 coordination.k8s.io/v1beta1 crd.k8s.amazonaws.com/v1alpha1 discovery.k8s.io/v1 discovery.k8s.io/v1beta1 events.k8s.io/v1 events.k8s.io/v1beta1 extensions/v1beta1 flowcontrol.apiserver.k8s.io/v1beta1 networking.k8s.io/v1 networking.k8s.io/v1beta1 node.k8s.io/v1 node.k8s.io/v1beta1 policy/v1 policy/v1beta1 rbac.authorization.k8s.io/v1 rbac.authorization.k8s.io/v1beta1 scheduling.k8s.io/v1 scheduling.k8s.io/v1beta1 secrets-store.csi.x-k8s.io/v1 secrets-store.csi.x-k8s.io/v1alpha1 storage.k8s.io/v1 storage.k8s.io/v1beta1 v1 vpcresources.k8s.aws/v1beta1 ```

So, it appears that my cluster supports rbac.authorization.k8s.io api version v1 and v1beta1, but not v1alpha1.

I'm really confused about why kubetest is failing to access my v1.21 cluster.

[timblaktu]

OK, wading my way through this, I see:

1. The error happens bc when populating the api_clients for ClusterRoleBinding class, the kubernetes client does not have a RbacAuthorizationV1alpha1Api member.
2. The kubernetes client comes from the kubernetes package. On my system I have v23.3.0 of this package.
3. Looking in kubernetes client v23.3.0 I see that it indeed only has RbacAuthorizationV1Api and no beta/alpha versions. THIS!

So, I think what's needed in my case is to downgrade the kubernetes package on my system.

Looking at the previous version of the kubernetes client package, I see that it indeed supports RbacAuthorizationV1alpha1Api as well as RbacAuthorizationV1Api.

EDIT: Ugh, I see that I cannot simply downgrade kubernetes client package because it is being installed as a dependency of kubetest. I will instead downgrade kubetest itself until I get a previous version of the kubernetes client that supports RbacAuthorizationV1alpha1Api.

[timblaktu]

Removing kubetest and reinstalling kubetest pinned to 0.9.4 and kubernetes pinned to 21.7.0 (the most recent version that appears to include v1alpha/beta1), I run into similar issue in DaemonSet Api versioning:

  File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/daemonset.py", line 17, in <module>
    class DaemonSet(ApiObject):
  File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/daemonset.py", line 35, in DaemonSet
    "apps/v1beta1": client.AppsV1beta1Api,
AttributeError: module 'kubernetes.client' has no attribute 'AppsV1beta1Api'

I believe this is what @tyxeron was referring to.

How do I find the version of kubetest (and accompanying kubernetes client package) that officially supports kubernetes v1.21?