Ramen controllers are caching secret/configmaps from all namespaces

[akalenyu]

Initally sparked by a casual conversation with @nirs about a bug around secret objects, I took a peek at the source code and it indeed looks like some controllers are caching all configmaps and secrets. This both:

• Aggressively increases memory usage (think: production cluster of an org with infinite CMs/Secrets)
• Poses a security threat with Ramen having access to all user sensitive data in the cluster - rbac practice reference

This can be solved by instead only caching the secrets/configmaps in the ramen namespace From a quick read, ramen only cares about those anyway https://github.com/RamenDR/ramen/blob/da2b47b0a7844cf791d9fa7cfd4953ab79251ede/controllers/drcluster_controller.go#L162-L165

This can be demonstrated in one of the default drenv environments (I used test/envs/regional-dr-kubevirt.yaml):

for i in {1..200}; do kubectl --context dr1 create cm test-cm-$i -n default --from-file=../manifests/largedatafile.txt ; done

$ kubectl get pods --context dr1 -n ramen-system -w
ramen-dr-cluster-operator-896d8c9f6-krbtd   2/2     Running             0             11s
ramen-dr-cluster-operator-896d8c9f6-krbtd   1/2     OOMKilled           0             34s
ramen-dr-cluster-operator-896d8c9f6-krbtd   1/2     Running             1 (1s ago)    35s
ramen-dr-cluster-operator-896d8c9f6-krbtd   2/2     Running             1 (7s ago)    41s
ramen-dr-cluster-operator-896d8c9f6-krbtd   1/2     OOMKilled           1 (31s ago)   65s

May have to kill the existing ramen pod. Could also watch the memory usage grow with minikube addons enable metrics-server --profile dr1 and then kubectl --context dr1 top pod -n ramen-system

See also

• https://bugzilla.redhat.com/2283489 Bug causing creation of up to 47,000 secrets consuming 329 MiB, leading to OOM in ramen and other operators.

[akalenyu]

/assign akalenyu