Closed motjuste closed 2 months ago
Thank you for reporting us your feedback!
The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5749.
This message was autogenerated
This issue is caused by juju v3.5.1, we have identified it already (see https://github.com/canonical/bundle-kubeflow/issues/921). The fix is already in juju v3.5.2, but hasn't been released yet. As a workaround, we are pinning the agent to 3.5.0 in some of our CIs (kfp and notebooks), I suggest you do it if you'd like to deploy kfp-operators using juju 3.5. Alternatively, you could use 3.1 (which is the supported version for CKF 1.8). 3.4 also works (v3.4.3, earlier versions had bugs), but we haven't fully tested it.
In the next two weeks we will define the juju supported version and make a communication, so for now please use juju 3.5 with caution.
Closing it as it is a duplicate of https://github.com/canonical/bundle-kubeflow/issues/921, please refer to that other issue for updates. Thanks @motjuste !
Thanks for the clarification, and glad we could help. It all works pretty well with Juju 3.3 for sure too. We'll be careful using Juju 3.5.
Bug Description
At Solutions QA, we are seeing all test runs for Charmed-Kubeflow failing when using Juju 3.5/candidate since at least 23-May-2024 where the deployment of the charm does not finish within 2 hours. We see multiple errors in the debug-log for Juju with multiple units reporting exceptions similar to the traceback below. We are actually seeing multiple units reporting similar
ops.pebble.PathError: permission-denied - cannot create directory: ...
for different directories.In the Juju status, we see that
kfp-persistence
stays stuck in waiting with message[container:persistenceagent] Waiting for Pebble services (persistenceagent). If this persists, it could be a blockin...
To Reproduce
We are deploying using FCE in a very standard manner.
Environment
In baremetal and AWS clouds, each with:
Relevant Log Output
Additional Context
This may potentially be related to this bug reported against Juju 3.5.0.
More identified recurrences of this bug at Solutions QA can be found here.