UATs cannot run in a remote node due to PVC hostpath

[orfeas-k]

Bug Description

UATs run by creating a job. This job creates a pod and a pvc which they mount to a local hostpath. That means that when the job and pod is scheduled to a remote node, where the uats repo is not available or it is in a different path than the one mounted, the UATs will fail, since the mounted path doesn't exist or it doesn't contain the expected files.

1. UATs will actually not fail but rather wait for the job to complete until waiting times out.
2. Looking at the pods, we see that the test-kubeflow-<hash> pod created, logs the following

   ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
   bash: line 2: pytest: command not found

To Reproduce

This can be reproduced by connecting to a remote cluster where CKF is deployed (e.g. AKS or EKS) and then trying to run kubeflow UATs (with tox -e kubeflow)

Environment

AKS/EKS Juju 3.4 tox 4.14.1

Relevant Log Output

ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
bash: line 2: pytest: command not found

Additional Context

No response

[syncronize-issues-to-jira[bot]]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5473.

This message was autogenerated

[orfeas-k]

Proposed solution

Using files in a volume mounted to a local path is not possible when a pod in a remote node needs to access those. Thus, we should enable UATs to be able run in two different modes

Run tests from local copy

This is essentially what we are doing right now. The job created runs the tests from a volume mounted to the local copy of charmed-kubeflow-uats repository, using hostPath.

Implementation

For any environment, the user needs to append -local at the end, which will function like we do for bundle tests.

tox -e <environment>-local

Run tests from a remote branch

(see next comment for update) In this case, the user pr for rovides the tox environment with the remote branch from which they 'd like to run tests from. Then, the job creates an emptyDir volume and spins an initContainer that syncs a volume's directory to the content of a charmed-kubeflow-uats , using kubernetes git-sync image .

Implementation

In this case, the user needs to append -remote at the end and provide a branch argument to pytest:

tox -e <environment>-remote -- --branch=<branch-name>  

[orfeas-k]

Proposed solution (update)

Run tests from remote commit

Instead of allowing the user to define the remote branch they wish to run tests from, tests will automatically grab the currently checked out commit of the local charmed-kubeflow-uats repository and use this to fetch the tests in the created volume. This way:

• the tests running will be in sync to what the user sees locally (unless changes are uncommited)
• the user will not have to specify the branch manually. That also means that tests will not silently run from main in the case the user forgets to provide a branch argument.
• the tests will fail if the checked out commit isn't pushed to the remote repository and will print git-sync initContainer logs.