search

k8ssandra / k8ssandra-operator

The Kubernetes operator for K8ssandra
https://k8ssandra.io/
Apache License 2.0
173 stars 79 forks source link

Encryption material creation is very slow #636

Open Miles-Garnsey opened 2 years ago

Miles-Garnsey commented 2 years ago

What happened?

When spinning up new clusters I'm seeing them stuck with the following error for quite some time:

MountVolume.SetUp failed for volume "encryption-cred-storage" : secret "dc1-keystore" not found

Did you expect to see something different?

Yes, I wouldn't expect encryption materials creation to be quite so slow, can we look at how this is sequenced so that it doesn't delay Cassandra pod creation?

┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: K8OP-197

jsanda commented 2 years ago

Can you provide some more details such as:

Miles-Garnsey commented 2 years ago

Sure.

Kubernetes distro: kind Storage class used (or even better a K8ssandraCluster manifest): Attached (I don't think storage class is relevant given this it is a secret which is taking a while to be created). Maybe kubelet logs if anything appears relevant: events attached. Going to each Kubelet seems like overkill, let me know what you're expecting to see?

I've measured the timings a bit more carefully, it looks like this issue slows the cluster start down by 1.5 - 2 minutes or so in most cases. It is noisy however, as we see the error occurring 7+ times.

This isn't huge next to the 14-15 minutes Cassandra frequently takes to come up, but it still seems like it might be a useful optimisation if we can move <DATACENTER_NAME>-keystore secret creation forward in the reconciliation cycle to avoid it causing any pause at all.

events.log

k8ssandra-cluster copy.txt