`[PVCs]` Create public documentation for data backups and restoration

DnPlas commented 2 months ago

Context

We need to create public documentation for users that want to backup their data and restore from it.

What needs to get done

Create a section in this discourse post with information about data backup and restoration for charms that use PVCs (mlmd and minio).

Definition of Done

A section in the discourse post that exposes information about backup and restoration for PVCs used in CKF.

syncronize-issues-to-jira[bot] commented 2 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5599.

This message was autogenerated

kimwnasptd commented 2 months ago

In the first iteration of the guide, it instructs users to have a config for rclone that sets the secret-key of MinIO to a value of minio123.

Our Charm though right now sets an empty value for the secret-key, which means that the MinIO Charm will create a random password and not reflect its value in the config option. https://github.com/canonical/minio-operator/blob/track/ckf-1.8/src/charm.py#L219-L234

This urged me to change the password in MinIO (although we dropped this approach, since it's bad UX to expect users to change the password to make a backup) but then stumbled upon the issue that the KFP Profile Operator will not correctly propagate the minio secret-key to the user namespaces https://github.com/canonical/kfp-operators/issues/449

The above resulted in the following errors in the UI and ml-pipeline-ui-artifact in the user namespace

Failed to get object in bucket: S3Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.

kimwnasptd commented 2 months ago

To rewrite the instructions we have and tell users which MinIO secret-key to use and where to get the current value from. Currently, if the config option is set to "" we don't have a direct way from the Charm to get this. Ideally in the future this should be a dedicated charm action https://github.com/canonical/minio-operator/issues/168

For now we'll use the following command for getting the secret-key value

juju show-unit kfp-ui/0 \
        | yq '.kfp-ui/0.relation-info.[] | select (.endpoint == "object-storage") | .application-data.data' \
        | yq '.secret-key'

Note that the above command is not targeting MinIO, which is the producer, but a consumer. This was because we couldn't find a way to get this outgoing value from MinIO.

kimwnasptd commented 2 months ago

Then, I've created some follow-up issues so that in the future we move this backup/restore logic to the Charms

canonical / bundle-kubeflow