Integrate the UATs into the Kubeflow bundle CI on self-hosted runners

[phoevos]

As a next step, we need to integrate the UATs provided as part of this repo into the bundle-kubeflow CI. These are the main concerns:

• Kubeflow must be already installed
• MLFlow must be already installed
• We need a way to specify versions of the tests to run for each supported version of the above (explore branching)

[phoevos]

Closing and reopening to trigger Jira automation.

[orfeas-k]

Configure Microk8s with proper DNS server

Rewriting the full-bundle-tests workflow to use Operator action and integrate the UATs, we stumbled on this error

Exception: Deploy failed with code: 1, 
stdout: , 
stderr ERROR cannot deploy bundle: cannot resolve charm or bundle "admission-webhook": resolving with preferred channel: Post "https://api.charmhub.io/v2/charms/refresh": dial tcp: lookup api.charmhub.io on 10.152.183.10:53: server misbehaving

10.152.183.10 is the IP of CoreDNS pod and looks like juju controller pod doesn't use the correct DNS server. The issue is that we enabled DNS addon here but without using the correct Addons string from here.

[orfeas-k]

kfp-api stuck in Waiting

Hit this issue while trying to deploy bundle (and run UATs) https://github.com/canonical/kfp-operators/issues/382

[orfeas-k]

Lightkube httpx 403 errors

Trying to run the UATs, we stumbled on the following error

  File "/home/ubuntu/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.10/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ProxyError: 403 Forbidden

Fortunately, @phoevos had resolved this in the past and documented that in https://github.com/canonical/charmed-kubeflow-uats/issues/27#issue-1908758148. Proposed solution is to tell lightkube (httpx) to ignore the ENV variables, so that the Proxy env vars are ignored. As indicated in the issue, this is done by adding trust_env=False when initializing a lightkube Client like this

-    lightkube_client = Client()
+    lightkube_client = Client(trust_env=False)

[orfeas-k]

Python version and requirements.txt file

Another issue with running the UATs on self-hosted runners is that requirements.txt have been produced using Python 3.8 while self-hosted come with Python 3.10 and this results in a conflicting dependency error

ERROR: Cannot install -r requirements.txt (line 64) and websockets==7.0 because these package versions have conflicting dependencies.

In order to resolve this, we can either:

1. Update requirements.txt using Python 3.10 and commit this to the UATs repo. The downside of this is that it will render UATs not "run-able" in environments with Python 3.8.
2. Set up Python 3.8 in self-hosted runners.

Since option 2 aligns with our development environment at the moment and has been already implemented and tested by @phoevos, we 'll go with that option.