GCP KMS - SCT support

[mykaul]

GCP KMS is not covered in SCT today (AFAIK) and is coming in 2024.2. We need to add, similar to AWS KMS, support to it.

[rayakurl]

@lersek @elcallio is there an ETA for this task?

[lersek]

@rayakurl this is at the bottom of my TODO list. I'm a beginner; I've been working through the "more approachable" items near the top of my TODO list (which are still hard).

I'll get to this one (I'm trying not to insert further (smaller) items near the top of my list), but I have no ETA. I don't know nearly enough to attack this problem at once, especially not concurrently with other assignments. Seasoned developers can multi-task well (speaking from experience), but it depends on lexical knowledge, and I'm not there yet; I have minimal background in scylla, for the time being.

[rjeczalik]

FWIW This is Terraform template for GCP KMS keys that cloud uses in its automation:

https://github.com/scylladb/siren-byok/blob/main/services/key-service/server/parts/gcp/key.yaml

I tested 2024.2.0~rc2-0.20240904.4c26004e5311 manually and it looks like it works.

For the GCP KeyRing and CryptoKey we are using defaults, and I am wondering if defaults are good enough or do we need to somehow harden them?

[lersek]

I guess I've formed a rudimentary / superficial idea about this ticket.

At commit 2e9e77c057ab, we have a number of encryption-at-rest tests ("jenkins files") in folder jenkins-pipelines/enterprise/SCT_Enterprise_Features/EncryptionAtRest/.

Each EaR-longevity-*.jenkinsfile there binds a test case (from under test-cases/longevity/) to an encryption-at-rest configuration generator (from under configurations/):

	hytrust-kmip	kms	local	replicated
encryption-at-rest-200GB-6h	x	x	x	x
50GB-3days-authorization-and-tls-ssl	x		x	x
encryption-at-rest-20GB-6h-multidc		x
encryption-at-rest-50GB-1day-authorization-and-tls-ssl		x

The AWS KMS generator is configurations/kms-ear.yaml; it's contents are (apparently) mostly explained here. The 'kms_host': 'auto' part is not valid for scylla.yaml directly, but its behavior is explained by commit a45b72164b60 ("feature(KMS): add KMS key rotation on test level", 2023-08-27).

Roughly speaking, AIUI, we should implement a "GCP key provider" column for the above table. scylla-enterprise already supports GCP key storage (ent/encryption/gcp_key_provider.*); that's what needs to be tested. Immediate questions:

• What rows in the above table should be covered?
• A big bunch of specific config values are needed for setting up GCP; where do we get those from?

[lersek]

(Note to self: the one SCT user/developer guide I'm aware of is at https://docs.google.com/presentation/d/1QC9Tf4mVvs6yql7QmqUvJqABX2-mW2FN8JAvnRIMbQM/edit.)

[rjeczalik]

Also there is an aspect of CMK key rotation - AWS rotates underlying key material each 365 days, GCP - each 90 days.

It would be also good to cover if everything is sound when we force the rotation manually (under load). The answer - whether key rotation affects or does not affect the cluster serving data - will allow us to design cloud operations accordingly (a bit related issue here: scylladb/siren-byok#449):

a. if it does not affect (no performance issues reloading the DEK etc) - then we can rely on auto-rotation, unifying the intervals on both clouds.

b. if it affects - then we need to rotate manually in a maintenance window and this requires development.

@mykaul @gmizrahi FYI