Provision S3 buckets on cloudscale with cloudscale-SDK

ccremer commented 2 years ago

Summary

As user\ I want to setup backups with S3 buckets on cloudscale.ch infrastructure\ So that I can backup my DB instances

Context

In #74 we discussed how provider-jet-cloudscale should be integrated to make use of automatic S3 bucket provisioning. In an internal discussion though, concerns were raised that this Crossplane integration, as well as provider-jet-cloudscale are not worth the added complexity vs directly provision buckets through cloud APIs and SDKs.

Thus we decided to implement bucket provisioning through code in the operator.

There are various terms that need to be defined:

cloudscale.ch project: a project is the scope where the cloudscale.ch API tokens operate at. It's where all Object Users for a given cloudscale.ch token will be provisioned. There's no higher abstraction available via API, so we can't provision a project per tenant unfortunately. A cloudscale.ch account can have multiple projects.
cloudscale.ch Object User: this represents an ACL and a set of credentials that can connect to cloudscale.ch's S3 API. By default the credentials associated with an Objects User can only create and access buckets for itself. It can't access buckets of other Object Users.
postgres instance: a single instance provisioned by our operator.
APPUiO Customer: a customer of APPUiO. Each customer can have multiple organizations.
APPUiO Organization: a unit that groups together Users, Teams and Namespaces. It’s the entity that owns the associated resources and the business connection. Billing is done on this level.

Each instance should get its own bucket and Objects User. This avoids locking issues with K8up/restic.

Also, according to cloudscale.ch, if we announce it early enough they can increase limits on the S3 servics for us. So sharding should not be necessary on our end.

drawio

Out of Scope

Crossplane integration
Any other S3 provider other than cloudscale.ch
Making bucket provisioning work in kind with Minio

Further links

Cloudscale Go SDK: https://github.com/cloudscale-ch/cloudscale-go-sdk
Cloudscale API docs: https://www.cloudscale.ch/en/api/v1
POC: #90

Acceptance Criteria

Given an APPUiO Organisation
When they provision their first AppCat instance
Then provision an Objects Users with the name equal to the APPUiO Organisation ID

Given a PostgresqlStandalone instance CRD
When spec.backup.enabled is true
And create an S3 bucket on cloudscale.ch through API
And store the access credentials in a Kubernetes Secret in the service namespace
And configure K8up backup schedule to use this bucket

Given a PostgresqlStandalone instance CR
And the S3 bucket credentials exist
When the instance gets deleted
Then delete all objects in the backup bucket through S3 client *
And delete the backup bucket on cloudscale.ch through API

Given an instance with enabled backup
When the user disables the backup
Then simply remove the K8up schedule
And DON'T delete the bucket
And DON'T delete the bucket secret

*If deleting a bucket isn't possible without deleting all data first.

Implementation Ideas

No response

ccremer commented 2 years ago

I created a PoC that provisions buckets with Minio, for local testing. It shows how relatively easy it is to create object users and buckets, and I expect that the cloudscale SDK should be similarly easy to use. https://github.com/vshn/appcat-service-postgresql/pull/90

Kidswiss commented 2 years ago

BTW: it doesn't look like cloudscale's SDK supports creating buckets. So for that we could re-use the logic to create the bucket from your POC.

ccremer commented 2 years ago

It's possible that we'll have to use a generic S3 client SDK to make buckets once an object user has been provisioned with a specific SDK. Maybe the Minio client does it as well, maybe there are others. What's missing in the PoC is potentially adding the ACL so that the user can actually read-write the bucket

Kidswiss commented 2 years ago

The minio-sdk can be used as a general purpose S3 client. It's what we use in K8up (https://github.com/k8up-io/k8up/blob/master/restic/s3/client.go) and restic (https://github.com/restic/restic/blob/master/internal/backend/s3/s3.go)

As for the ACL: do you talk about the ACLs for a minio instance, or for cloudscale's S3? Because the latter sets up some ACLs already that give the object-users permissions to their own buckets.

ccremer commented 2 years ago

As for the ACL: do you talk about the ACLs for a minio instance, or for cloudscale's S3? Because the latter sets up some ACLs already that give the object-users permissions to their own buckets.

ACLs for cloudscale's S3. But you just answered my question :)

Kidswiss commented 2 years ago

I've also added the deletion of the bucket. For the first implementation we just remove the bucket as soon as the instance is deleted. We can add delays and deletion protection in further iterations.

vshn / appcat-service-postgresql