Closed goat-ssh closed 1 day ago
Script to patch:
#!/usr/bin/env bash
##set -ex
for namespace in $(kubectl get cratedb --all-namespaces -o jsonpath='{.items[*].metadata.namespace}'); do
depl=$(kubectl -n "$namespace" get deployment --selector=app.kubernetes.io/component=grand-central -o jsonpath='{range .items[*]}{.metadata.name}{end}')
echo "Processing Namespace: $namespace and GC: $depl"
# patch the deployment to update resources
kubectl -n "$namespace" patch deployment "$depl" -p'{"spec":{"template":{"spec":{"containers":[{"name":"grand-central-api","resources":{"requests":{"cpu":"64m","memory":"256Mi"},"limits":{"cpu":"500m","memory":"256Mi"}}}]}}}}'
done
Progress:
dev:
prod:
How to check current settings:
kubectl get deploy -l app.kubernetes.io/component=grand-central -o=jsonpath='{range .items[*]}{.metadata.name} REQ:{..containers..resources.requests} LIMIT:{..containers..resources.limits}{"\n"}{end}' -A
Summary of changes
We continue to see occasional crash due to OOMs especially on clusters with frequent scheduled jobs. Also reduce CPU limit to 500m as we're too generous with 2 CPUs now.
Checklist
CHANGES.rst