crate / crate-operator

The CrateDB Kubernetes Operator provides a convenient way to run CrateDB clusters inside Kubernetes.
https://crate.io
Apache License 2.0
22 stars 6 forks source link

GC: increase memory to 256Mi and drop CPU limit to 500m #625

Closed goat-ssh closed 1 day ago

goat-ssh commented 5 days ago

Summary of changes

We continue to see occasional crash due to OOMs especially on clusters with frequent scheduled jobs. Also reduce CPU limit to 500m as we're too generous with 2 CPUs now.

Checklist

goat-ssh commented 1 day ago

Script to patch:

#!/usr/bin/env bash
##set -ex

for namespace in $(kubectl get cratedb --all-namespaces -o jsonpath='{.items[*].metadata.namespace}'); do
  depl=$(kubectl -n "$namespace" get deployment --selector=app.kubernetes.io/component=grand-central -o jsonpath='{range .items[*]}{.metadata.name}{end}')
  echo "Processing Namespace: $namespace and GC: $depl"
  # patch the deployment to update resources
  kubectl -n "$namespace" patch deployment "$depl" -p'{"spec":{"template":{"spec":{"containers":[{"name":"grand-central-api","resources":{"requests":{"cpu":"64m","memory":"256Mi"},"limits":{"cpu":"500m","memory":"256Mi"}}}]}}}}'
done
goat-ssh commented 1 day ago

Progress:

dev:

prod:

goat-ssh commented 1 day ago

How to check current settings:

kubectl get deploy -l app.kubernetes.io/component=grand-central -o=jsonpath='{range .items[*]}{.metadata.name} REQ:{..containers..resources.requests} LIMIT:{..containers..resources.limits}{"\n"}{end}' -A