`mysql-k8s` backup unexpected behaviour

[DnPlas]

Steps to reproduce

kfp-db = mysql-k8s

1. Setup access and secret keys

juju run s3-integrator/leader sync-s3-credentials \
    access-key=<access-key> \
    secret-key=<secret-key>

2. Configure the s3 integrator

juju config s3-integrator endpoint=<s3 endpoint> \
    bucket="daniela-bucket-1" \ # <-- this is the name of my bucket
    path="backup-folder" \ # <--this is an empty folder I created in my s3 a priori
    region=eu-west-2

3. List backups fails

juju run kfp-db/leader list-backups 

Running operation 38 with 1 task
  - task 39 on unit-kfp-db-0

Waiting for task 39...
Action id 39 failed: Failed to retrieve backup ids from S3

4. Created a backup

juju run kfp-db/1 create-backup

This created a new folder in my s3 named daniela-bucket-1 (same as the name I passed in the config), then inside it, there is another folder named backup-folder (same as the name I passed in the config). Inside backup-folder I see backup.log and metadata files.

I would not expect the action to create a new folder(s) in my s3 bucket.

The tree in my s3 looks like this: Amazon S3 > Buckets > daniela-bucket-1 > daniela-bucket-1/ > backup-folder

5. List backups still fails, even if I can see backups in my s3:

juju run kfp-db/leader list-backups 
Running operation 40 with 1 task
  - task 41 on unit-kfp-db-0

Waiting for task 41...
Action id 41 failed: Failed to retrieve backup ids from S3

juju debug-log

unit-kfp-db-0: 15:59:10 INFO unit.kfp-db/0.juju-log Listing subdirectories from S3 bucket=daniela-bucket-1, path=backup-folder
unit-kfp-db-0: 15:59:11 ERROR unit.kfp-db/0.juju-log Failed to list subdirectories in S3 bucket=daniela-bucket-1, path=backup-folder
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/mysql/v0/s3_helpers.py", line 145, in list_backups_in_s3_path
    for page in list_objects_v2_paginator.paginate(
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/botocore/paginate.py", line 269, in __iter__
    response = self._make_request(current_kwargs)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/botocore/paginate.py", line 357, in _make_request
    return self._method(**current_kwargs)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: The specified key does not exist.
unit-kfp-db-0: 15:59:11 ERROR unit.kfp-db/0.juju-log Failed to retrieve backup ids from S3

Expected behavior

1. The backup action does not create new folders in the s3 bucket
2. I can list available backups in the s3 storage

Actual behavior

1. The backup action creates new folders in my s3
2. list-backups returns an error (see above)

Versions

Operating system: ubuntu 22.04

Juju CLI: 3.1/stable

Juju agent: 3.1

Charm revision: 8.0/stable

microk8s: 1.25-strict/stable

Log output

Added above.

[github-actions[bot]]

https://warthogs.atlassian.net/browse/DPE-4218

[shayancanonical]

confirmed that the provided endpoint had the bucket name which caused the above issue: https://<bucket-name>.s3.<region-name>.amazonaws.com was used https://s3.<region-name>.amazonaws.com should be used

the below is output after the corrected endpoint is used:

(venv) ubuntu@localhost:~$ juju run mysql/leader list-backups 
Running operation 21 with 1 task
  - task 22 on unit-mysql-0

Waiting for task 22...
backups: |-
  backup-id             | backup-type  | backup-status
  ----------------------------------------------------

(venv) ubuntu@localhost:~$ juju run mysql/leader create-backup
Running operation 23 with 1 task
  - task 24 on unit-mysql-0

Waiting for task 24...
backup-id: "2024-04-25T16:05:39Z"

(venv) ubuntu@localhost:~$ juju run mysql/leader list-backups 
Running operation 25 with 1 task
  - task 26 on unit-mysql-0

Waiting for task 26...
backups: |-
  backup-id             | backup-type  | backup-status
  ----------------------------------------------------
  2024-04-25T16:05:39Z  | physical     | finished

[DnPlas]

I can confirm that the commands succeed after fixing the endpoint, thanks! Closing this as there is no issue with the charm.