AssertionError: Katib Experiment was not successful

mvlassis commented 3 months ago

Bug Description

This issue was encountered in the deploy-cfk-to-eks (1.8) action in bundle-kubeflow repository. The full logs can be found here.

The katib-integration test in test_notebook.py fails and raises an AssertionError. This is the relevant log call from the logs:

-------------------------------- live log call ---------------------------------
INFO     test_notebooks:test_notebooks.py:44 Running katib-integration.ipynb...
ERROR    test_notebooks:test_notebooks.py:58 Cell In[8], line 8, in assert_experiment_succeeded(client, experiment)
      1 @retry(
      2     wait=wait_exponential(multiplier=2, min=1, max=10),
      3     stop=stop_after_attempt(30),
      4     reraise=True,
      5 )
      6 def assert_experiment_succeeded(client, experiment):
      7     """Wait for the Katib Experiment to complete successfully."""
----> 8     assert client.is_experiment_succeeded(name=experiment), f"Katib Experiment was not successful."
AssertionError: Katib Experiment was not successful.
FAILED

Because the error was encountered during a Github action, I couldn't access the deployment and investigate further.

Note that this issue was not encountered during a previous run of the Github action, which can be found here. It's not clear whether this issue is reproducible or just intermittent.

To Reproduce

From the main page of the bundle-kubeflow repository, go to Actions, select the "Create EKS cluster, deploy CKF and run bundle test" action, and run it with the following options:

Comma-separated list of bundle versions e.g. "1.7","1.8": 1.8
Kubernetes version to be used for the AKS cluster: Leave empty
Branch to run the UATs from e.g. main or track/1.8: Leave empty

Environment

This job tries to deploy the UATs, using the following configuration from the dependencies.yaml file found here:

K8S_VERSION: "1.29"
JUJU_VERSION: "3.4"
JUJU_VERSION_WITH_PATCH: "3.4.4"
UATS_BRANCH: "track/1.8"

Relevant Log Output

-------------------------------- live log call ---------------------------------
INFO     test_notebooks:test_notebooks.py:44 Running katib-integration.ipynb...
ERROR    test_notebooks:test_notebooks.py:58 Cell In[8], line 8, in assert_experiment_succeeded(client, experiment)
      1 @retry(
      2     wait=wait_exponential(multiplier=2, min=1, max=10),
      3     stop=stop_after_attempt(30),
      4     reraise=True,
      5 )
      6 def assert_experiment_succeeded(client, experiment):
      7     """Wait for the Katib Experiment to complete successfully."""
----> 8     assert client.is_experiment_succeeded(name=experiment), f"Katib Experiment was not successful."
AssertionError: Katib Experiment was not successful.
FAILED