search

canonical / kfp-operators

Kubeflow Pipelines Operators
Apache License 2.0
2 stars 12 forks source link

KFP Profiles Operator doesn't update MinIO's credentials if updated in MinIO's config #449

Open kimwnasptd opened 5 months ago

kimwnasptd commented 5 months ago

Bug Description

When we change the MinIO secret-key config option, the KFP Profiles Operator will not

  1. Update the mlpipeline-minio-artifact K8s Secret, in the user namespace
  2. Restart the ml-pipeline-ui-artifact Deployment's pod, to get the new value for the secret

The above results in seeing the following error in the KFP UI, when trying to download the artifacts of an already finished Pipeline (that live in S3)

Failed to get object in bucket: S3Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.

These logs are coming from the ml-pipeline-ui-artifact pod in the user's namespace, which is responsible for fetching the minio artifacts. But because now MinIO has different credentials and that pod has still the outdated once it then fails to get the artifacts

To Reproduce

  1. Deploy CKF 1.8
  2. Run the data-passing pipeline, which creates some artifacts in MinIO
  3. juju config minio secret-key=minio123
  4. Go to the KFP UI and click on the step that produced a Dataset or Model artifact and try to download them from the minio link in the UI

To confirm the user namespaced MinIO credentials are incorrect you can do:

  1. kubectl get secret -n admin mlpipeline-minio-artifact -o yaml
  2. Decode the hash, and it'll be a radom value (which is the initial secret-key since the config option is initially empty)

Environment

CKF 1.8

Relevant Log Output

# From the `ml-pipeline-ui-artifact` pod in the user namespace:
Failed to get object in bucket: S3Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.


### Additional Context

_No response_
syncronize-issues-to-jira[bot] commented 5 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5621.

This message was autogenerated