Velero AWS plugin should support Tencent COS virtual-styled domain

[quangthe]

What steps did you take and what happened:

Install velero on Tencent TKE cluster: https://velero.io/docs/main/contributions/tencent-config/.

velero install  --provider aws --plugins velero/velero-plugin-for-aws:v1.2.1 --bucket <bucket-name> \
--secret-file ./credentials-velero \
--use-node-agent \
--default-volumes-to-fs-backup \
--backup-location-config \
region=ap-shanghai,s3ForcePathStyle="true",s3Url=https://cos.ap-shanghai.myqcloud.com

Get backup location

velero backup-location get

Output:

NAME      PROVIDER   BUCKET/PREFIX            PHASE         LAST VALIDATED                  ACCESS MODE   DEFAULT
default   aws        prod-velero-1259416121   Unavailable   2024-04-04 13:09:15 +0700 +07   ReadWrite     true

Velero logs

time="2024-04-04T04:02:14Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:152"
time="2024-04-04T04:02:14Z" level=error msg="fail to validate backup store" backup-storage-location=velero/default controller=backup-storage-location error="rpc error: code = Unknown desc = PathStyleDomainForbidden: The bucket you are attempting to access must be addressed using COS virtual-styled domain.\n\tstatus code: 403, request id: NjYwZTI2NDZfMzMxMGU3MDlfMTAyYmFfNWU2MzM3Yg==, host id: " error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:198" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).IsValid" logSource="pkg/controller/backup_storage_location_controller.go:155"
time="2024-04-04T04:02:14Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:131"
time="2024-04-04T04:02:14Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = PathStyleDomainForbidden: The bucket you are attempting to access must be addressed using COS virtual-styled domain.\n\tstatus code: 403, request id: NjYwZTI2NDZfMzMxMGU3MDlfMTAyYmFfNWU2MzM3Yg==, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"

What did you expect to happen: Backup location should be available.

Anything else you would like to add: Tencent COS Bucket Domain: https://www.tencentcloud.com/document/product/436/57456

For buckets created after January 1, 2024, COS will not support the use of path-style domain.

Domain Name Format	Example
path-style	cos.<Region>.myqcloud.com/<BucketName-APPID>
virtual-hosted-style	<BucketName-APPID>.cos.<Region>.myqcloud.com

Environment:

• Velero version (use velero version): v1.12.1
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): 1.26
• Cloud provider or hardware configuration: Tencent

[reasonerjt]

@quangthe per my undertstanding, this is customized for Tencent COS only.

This can be done by supporting additional flag in the BSL configuration, but I'm reluctant to put it in upstream, b/c I don't see we will have resource to test against tencent.

Therefore, in short term, I suggest you fork the plugin and implemented in the downstream.

[kaovilai]

@reasonerjt AWS also has virtual hosting. https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html It might also be possible to do this for minio.

[reasonerjt]

@reasonerjt AWS also has virtual hosting.

Yes and that's the default setting for AWS SDK

[quangthe]

Hi @reasonerjt

Set s3ForcePathStyle="false" seems working for Backup Location.

velero install  --provider aws --plugins velero/velero-plugin-for-aws:v1.2.1 --bucket <bukcet> \
--secret-file ./credentials-velero \
--use-node-agent \
--default-volumes-to-fs-backup \
--backup-location-config \
region=ap-shanghai,s3ForcePathStyle="false",s3Url=https://cos.ap-shanghai.myqcloud.com

BackupStorageLocations: default looks like this

spec:
  config:
    region: ap-shanghai
    s3ForcePathStyle: 'false'
    s3Url: https://cos.ap-shanghai.myqcloud.com/
  default: true
  objectStorage:
    bucket: <bucket>
  provider: aws

But we have another issue with FSB kopia/restic integration.

The BackupRepository (created by velero) still use the path-styled URL which is rejected by Tencent COS.

apiVersion: velero.io/v1
kind: BackupRepository
metadata:
  creationTimestamp: '2024-04-22T08:17:56Z'
  generateName: keycloak-default-restic-
  generation: 5
  labels:
    velero.io/repository-type: restic
    velero.io/storage-location: default
    velero.io/volume-namespace: keycloak
  name: keycloak-default-restic-8j9qm
  namespace: velero
spec:
  backupStorageLocation: default
  maintenanceFrequency: 168h0m0s
  repositoryType: restic
  resticIdentifier: >-
    s3:https://cos.ap-shanghai.myqcloud.com/<bucket>/restic/keycloak
status:
  message: >-
    error running command=restic init
    --repo=s3:https://cos.ap-shanghai.myqcloud.com/<bucket>/restic/keycloak
    --password-file=/tmp/credentials/velero/velero-repo-credentials-repository-password
    --cache-dir=/scratch/.cache/restic, stdout=, stderr=Fatal: create repository
    at
    s3:https://cos.ap-shanghai.myqcloud.com/<bucket>/restic/keycloak
    failed: client.BucketExists: The bucket you are attempting to access must be
    addressed using COS virtual-styled domain.

    : exit status 1
  phase: NotReady

[Im-oldxu]

@quangthe Hello, you from Tencent， The bucket you are attempting to access must be addressed using COS virtual-styled domain he BackupRepository (created by velero) still use the path-styled URL which is rejected by Tencent COS. Is it resolved