Lower memory usage of commons-operator

[lfrancke]

Currently the memory required by the commons-operator scales with the amount of Pods in a K8s cluster. This can lead to more or less unbounded memory usage.

We need to investigate how we can limit this. One possibility to limits this is by leveraging the work that will be done in https://github.com/stackabletech/issues/issues/188

[nightkr]

After some investigation, this largely comes down to two sources:

1. https://github.com/kube-rs/kube/issues/1209
2. We cache a lot of data about each Pod that we don't really need, we could whittle that down during the data fetch/watch rather than keep it around indefinitely (related to https://github.com/kube-rs/kube/issues/1210, but we could go much further)

[clux]

There are some docs on this now from kube in https://kube.rs/controllers/optimization/. in particular;

• pruning techniques (for say managedFields or .status store contributions)
• using metadata_watcher (also limits IO)
• pagination (now default)

should all help reduce the memory contribution of cached objects - although it will still generally scale with num_objects in your stores.

[nightkr]

• https://github.com/stackabletech/commons-operator/issues/237

[sbernauer]

kube 0.92.0 also seems to have nice memory improvements: https://kube.rs/blog/2024/06/11/watcher-memory-improvements/ :)