[velero-plugin-for-aws] Server Side Encryption with Customer provided keys fails for s3 server with self-signed ca

[arteonprifti]

What steps did you take and what happened: When trying to back up to a self-signed s3 server using the sse-c and giving the caCert the backup fails with

InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must be made over a secure connection.\n\tstatus code: 400

This seems to indicate that the server is not being called using https Trying without sse-c, the backup is successful. The same issue happens on v1.8.2 and v1.9.2 as well, with the same error.

Config:

    backupStorageLocation:
    - name: my-s3-location
      provider: aws
      default: true
      accessMode: ReadWrite
      caCert: ENTER-base64-CA
      config:
        region: ENTER-Region
        s3Url: https://ENTER-s3-url-here
        s3ForcePathStyle: "true"
        serverSideEncryption: AES256
        customerKeyEncryptionFile: "/credentials/backup_key"

• kubectl logs deployment/velero -n velero

  velero time="2024-05-29T16:19:57Z" level=info msg="hookTracker: map[], hookAttempted: 0, hookFailed: 0" backup=velero/nginx-backup-5 logSource="pkg/backup/backup.go:436"
  velero time="2024-05-29T16:19:57Z" level=info msg="Summary for skipped PVs: []" backup=velero/nginx-backup-5 logSource="pkg/backup/backup.go:445"
  velero time="2024-05-29T16:19:57Z" level=info msg="Backed up a total of 41 items" backup=velero/nginx-backup-5 logSource="pkg/backup/backup.go:449" progress=
  velero time="2024-05-29T16:19:57Z" level=info msg="Setting up backup store to persist the backup" backup=velero/nginx-backup-5 logSource="pkg/controller/backup_controller.go:729"
  velero time="2024-05-29T16:19:57Z" level=error msg="Error uploading log file" backup=nginx-backup-5 bucket=infra-ag-s01-velero error="rpc error: code = Unknown desc = error putting object backups/nginx-backup-5/ngi
  nx-backup-5-logs.gz: InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must be made over a secure connection.\n\tstatus code: 400, request id: 34742a9a-01e2-12db-b6eb-58a2e18df
  634, host id: " error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:356" error.function="main.(*ObjectStore).PutObject" logSource="pkg/persistence/object_store.go:252" prefix=
  velero time="2024-05-29T16:19:57Z" level=info msg="Initial backup processing complete, moving to Finalizing" backup=velero/nginx-backup-5 logSource="pkg/controller/backup_controller.go:743"
  velero time="2024-05-29T16:19:57Z" level=error msg="backup failed" backuprequest=velero/nginx-backup-5 controller=backup error="rpc error: code = Unknown desc = error putting object backups/nginx-backup-5/velero-ba
  ckup.json: InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must be made over a secure connection.\n\tstatus code: 400, request id: 65ae0437-e8e9-12bc-8e67-e8ebd3a387e2, host
  id: " logSource="pkg/controller/backup_controller.go:288"
  velero time="2024-05-29T16:19:57Z" level=info msg="Updating backup's final status" backuprequest=velero/nginx-backup-5 controller=backup logSource="pkg/controller/backup_controller.go:307"
  velero time="2024-05-29T16:20:21Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/my-s3-location controller=backup-storage-location logSource="pkg/controller/backup_storage_location_control
  ler.go:141"

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml

  
  ❯ velero backup describe nginx-backup-5
  Name:         nginx-backup-5
  Namespace:    velero
  Labels:       velero.io/storage-location=ionos
  Annotations:  velero.io/resource-timeout=10m0s
            velero.io/source-cluster-k8s-gitversion=v1.28.6
            velero.io/source-cluster-k8s-major-version=1
            velero.io/source-cluster-k8s-minor-version=28

Phase: Failed (run velero backup logs nginx-backup-5 for more information)

Namespaces: Included: ingress-nginx Excluded:

Resources: Included: * Excluded: Cluster-scoped: auto

Label selector:

**Environment:**

- Velero version (use `velero version`): v1.13.2
- Velero features (use `velero client config get features`): velero-plugin-for-aws
- Kubernetes version (use `kubectl version`):v1.28.6

**Vote on this issue!**

This is an invitation to the Velero community to vote on issues, you can see the project's [top voted issues listed here](https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).  
Use the "reaction smiley face" up to the right of this comment to vote.

- :+1: for "I would like to see this bug fixed as soon as possible"
- :-1: for "There are more important bugs to focus on right now"

[blackpiglet]

To me, this is related to how to use the SSE-c with the on-premise environment. What is the OSS backend in your environment?

I only find MinIO related document https://min.io/docs/minio/macos/administration/server-side-encryption/server-side-encryption-sse-c.html.

[arteonprifti]

Indeed, this was because of a misconfiguration of the s3 on premise. After fixing it, the error is gone. Thanks for the help