ca-scribner
commented
1 year ago We have an integration test that should confirm that changes to minio's secrets make their way into the kfp-profile-controller charm, so likely the issue is we aren't updating the credentials in the actual workload we deploy that is used by the metacontroller to deploy these secrets to users. We pass the secret to the workload via an environment parameter, so to change it we'd need to restart that Pod. First guess is that the charm is not restarting the workload Pod on minio secret change.
Would be an interesting first check to relate kfp-profile-controller to minio, change the secret, and see if the kfp-profile-controller's sync.py pod restarts
Barteus
Summary: When reconfiguring the access-key and secret-key to MinIO, after setting it wrongly ( less than 8 characters), the pipeline fails on the first step. The code of the step finishes successfully (Pod is Compeleted) but pipeline still fails..
Reproduce: Install charmed kubeflow using Quick start guide with minio credentails (access and secret): admin/admin Wait for juju model to settle. The minio is in the error state. Fix the credentials - use more than 8 characters for each config option (ie. minio123) Model turns green, but the values in the user namespaces (including admin) in the secret mlpipeline-minio-artifact are not the base64 of minio123, but still admin in accesskey and some random values in secretkey. Running the pipelines or accessing the artifacts ends with the result of an incorrect/unknown Access Key.
Expected behaviour: Minio credentails are synchronized with every namespace, because they were created there and are managed by Juju. This includes all CKF components in the kubeflow namespace and secrets in the user namespaces created by CKF.
Workaround: Manually access all Data Scientist namespaces and update the value of the secret. This procedure is manual and error prone.