Backup to Google Cloud Storage

filiprafaj commented 2 years ago

Hello, I am trying and failing to store backups to Google Cloud Storage.

I have set up Workload Identity to give the k8s service account the permissions to access the bucket.

I have also tried defining the _AWS_ACCESS_KEYID and _AWS_SECRET_ACCESSKEY variables.

Still I get Error: failed to take a full dump: failed to put dump.tar: operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

Is there anybody who made backups to GCS working?

Thank you!

ymmt2005 commented 2 years ago

Backup to GCS is not supported now. It can be added by implementing Bucket interface for GCS in this package. https://github.com/cybozu-go/moco/tree/main/pkg/bucket

We welcome a pull request for adding GCS support.

d-kuro commented 2 years ago

@filiprafaj GCS supports the S3 compatibility API. (Sorry, I have not verified this.) Could you please refer to the documentation and try again?

refs:

If it still doesn't work, it would be helpful if you could report it again, including the definition of BackupPolicy 🙏 https://cybozu-go.github.io/moco/usage.html#backuppolicy

filiprafaj commented 2 years ago

Hi @d-kuro , I have tried now with interoperability credentials and I am getting:

Error: failed to take a full dump: failed to put dump.tar: operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method. failed to take a full dump: failed to put dump.tar: operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

the backup file looks like this:

apiVersion: moco.cybozu.com/v1beta2
kind: BackupPolicy
metadata:
  namespace: default
  name: daily
spec:
  schedule: "@daily"
  jobConfig:
    serviceAccountName: moco-test-mysqlcluster
    env:
    - name: AWS_ACCESS_KEY_ID
      value: ***
    - name: AWS_SECRET_ACCESS_KEY
      value: ***
    bucketConfig:
      bucketName: ***
      endpointURL: https://storage.googleapis.com
    workVolume:
      emptyDir: {}
    memory: 1Gi
    maxMemory: 1Gi
    threads: 1

Prakharkarsh1 commented 2 years ago

Hii @filiprafaj i want to contribute to this issue

ymmt2005 commented 2 years ago

@Prakharkarsh1 Hi, Thank you for your intention to contribute to this project. We will review your pull request when it's ready.

yamatcha commented 2 years ago

@Prakharkarsh1

MOCO uses aws-sdk-go-v2 to connect to s3-compatible storage. However, aws-sdk-go-v2 is not compatible with third-party platforms and therefore cannot connect GCS. https://github.com/aws/aws-sdk-go-v2/issues/1816 So it would be better to implement for gcs bucket in moco/pkg/bucket/gcs.

ymmt2005 commented 2 years ago

We use minio to test S3 bucket implementation.

Likewise, we may use these tools to test GCS bucket implementation.

ymmt2005 commented 2 years ago

@Prakharkarsh1 Hello, Do you still want to contribute to this feature?

masa213f commented 1 year ago

Released https://github.com/cybozu-go/moco/releases/tag/v0.16.1

cybozu-go / moco

Backup to Google Cloud Storage #427