Upgrade between patch versions is broken

[Ebaneck]

Component:

'kubernetes', 'salt'

What happened:

A typical user is unable to upgrade between MetalK8s patch versions

What was expected:

I should be capable of upgrading from one patch version to another. In my case from MetalK8s 2.4.1 to 2.4.2

Steps to reproduce

• Bootstrap a platform with MetalK8s 2.4.1(a patch version)
• Run an upgrade to Metalk8s 2.4.2
• And boom upgrade will crash with the following report

[ERROR   ] Failed to import utils kubernetes_utils, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1534, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/var/cache/salt/master/extmods/utils/kubernetes_utils.py", line 296, in <module>
    model=k8s_client.ExtensionsV1beta1Ingress,
AttributeError: 'module' object has no attribute 'ExtensionsV1beta1Ingress'
[WARNING ] org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[CRITICAL] Specified ext_pillar interface metalk8s_endpoints is unavailable
[ERROR   ] Failed to import utils kubernetes_utils, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1534, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/var/cache/salt/master/extmods/utils/kubernetes_utils.py", line 296, in <module>
    model=k8s_client.ExtensionsV1beta1Ingress,
AttributeError: 'module' object has no attribute 'ExtensionsV1beta1Ingress'
[WARNING ] org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
Exception occurred in runner state.orchestrate_single: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/client/mixins.py", line 387, in _low
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/runners/state.py", line 169, in orchestrate_single
    __jid_event__.fire_event({'data': ret, 'outputter': 'highstate'}, 'progress')
ReferenceError: weakly-referenced object no longer exists
 done [23s]
> Performing Pre-Upgrade checks...[ERROR   ] Failed to import utils kubernetes_utils, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1534, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/var/cache/salt/master/extmods/utils/kubernetes_utils.py", line 296, in <module>
    model=k8s_client.ExtensionsV1beta1Ingress,
AttributeError: 'module' object has no attribute 'ExtensionsV1beta1Ingress'
[WARNING ] org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[CRITICAL] Specified ext_pillar interface metalk8s_endpoints is unavailable
[ERROR   ] Failed to import utils kubernetes_utils, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1534, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/var/cache/salt/master/extmods/utils/kubernetes_utils.py", line 296, in <module>
    model=k8s_client.ExtensionsV1beta1Ingress,
AttributeError: 'module' object has no attribute 'ExtensionsV1beta1Ingress'

Resolution proposal (optional): TBD

[TeddyAndrieux]

For me we need our custom module/utils to handle both version of python-kubernetes for the moment 8.0 and 10.0.

In the futur with #2029 I think we will need our salt module/utils to be compatible with at least one version before

[gdemonet]

Maybe it's a case where we need to upgrade the client before attempting to use it? It would make sense to have "the good version" in our Salt master

[TeddyAndrieux]

Or we can upgrade bootstrap stuff before doing anything else (meaning we upgrade bootstrap before patching the namespace version and running the upgrade pre-check states)