velero‘s restic daemonset cannot support Virtual-hosted–style

[cwyj]

What steps did you take and what happened: i try to backup pvc volume to TencentCloud cos by velero --use-restic，but path-style url is used by restic default and TencentCloud support Virtual-hosted–style only. Restic use -o s3.bucket-lookup=dns support Virtual-hosted–style(https://github.com/restic/restic/pull/2535)，but i cannot set option for velero‘s restic daemonset. velero‘s restic daemonset start with /velero restic server --features=

What did you expect to happen: velero‘s restic daemonset support Virtual-hosted–style url.

The following information will help us better understand what's going on:

=

Environment:

• Velero version (use velero version): Starting Velero restic server v1.7.1
• Velero features (use velero client config get features): features:
• Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
• OS (e.g. from /etc/os-release): tencent linux 2.6

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[reasonerjt]

First, we'll see whether in kopia integration we can fix it.

For Restic, I don't think current repoID has such information to help restic? If that's the case, we'll first try to fix the kopia part leveraging the storage-config or information in BSL.

[cwyj]

i have resolved this problem by replacing restic in velero image. write a shell script named restic like "/usr/bin/restic2 -o s3.bucket-lookup=dns -o s3.region=ap-guangzhou $*". /usr/bin/restic2 is original binary executable file in velero image.

[reasonerjt]

Thanks @cwyj but let me re-open this one as we are using different backend for fs based B/R

[Lyndon-Li]

Both Restic and Kopia are using minio-go client to connect S3 compatible object storage.

When making a connection, minio-go provides an option for caller to specify the lookup method:

minio.BucketLookupAuto
minio.BucketLookupDNS
minio.BucketLookupPath

At present Kopia doesn't support this itself, so when using Kopia, we don't have a way to specify the method. We can open an issue to Kopia for a fix.

Restic supports this by specifying the s3.bucket-lookup option. Therefore, in order to fix this problem for Restic path, Velero need to specify this option during calling Restic CLI.

[Lyndon-Li]

As a matter of fact, even if we have this support from the underlying repository, Velero needs to do one more thing: Velero cannot decide the option for users, so Velero needs to expose a new option in its CLI to ask for user's selection. Then Velero could deliver the selection to the underlying repository.

[princeteng]

As a matter of fact, even if we have this support from the underlying repository, Velero needs to do one more thing: Velero cannot decide the option for users, so Velero needs to expose a new option in its CLI to ask for user's selection. Then Velero could deliver the selection to the underlying repository.

Does velero have plans to fix this? Some object storage such as oss(product of alibaba) and tos(product of bytedance) no longer support path style, but only support virtual hosted style

[Lyndon-Li]

Let me conclude the situation after some more checks:

1. Both Kopia and Restic by default don't set the BucketLookup option
2. Restic gives users option to change the default option, but Kopia does not
3. If the BucketLookup option is not set, minio-go will use BucketLookupAuto
4. BucketLookupAuto means: for some selected cloud providers, minio-go chooses virtual host style, for others it uses path style
5. The selected providers are hard code in minio-go code as below, so only AWS S3, GCP and Aliyun OSS are set to use virtual host style:

   func IsVirtualHostSupported(endpointURL url.URL, bucketName string) bool {
   if endpointURL == sentinelURL {
       return false
   }
   // bucketName can be valid but '.' in the hostname will fail SSL
   // certificate validation. So do not use host-style for such buckets.
   if endpointURL.Scheme == "https" && strings.Contains(bucketName, ".") {
       return false
   }
   // Return true for all other cases
   return IsAmazonEndpoint(endpointURL) || IsGoogleEndpoint(endpointURL) || IsAliyunOSSEndpoint(endpointURL)
   }

[Lyndon-Li]

Therefore, if the cloud provider has already been in the list of minio-go's IsVirtualHostSupported, I believe Velero, Restic and Kopia will work with no problem.

[princeteng]

Therefore, if the cloud provider has already been in the list of minio-go's IsVirtualHostSupported, I believe Velero, Restic and Kopia will work with no problem.

Thank you very much, your answer is very clear, I try to modify the source code and recompile the velero image and binary files to deal with those providers that are not in the minio-go list.