Open chaosi-zju opened 5 months ago
/help
@chaosi-zju: This request has been marked as needing help from a contributor.
Please ensure the request meets the requirements listed here.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help
command.
It seems that inuse_space of eg1 and alloc_space of eg1 are the same picture
It seems CacheReader.Get
and CacheReader.List
double memory usage. For CacheReader.List
and CacheReader.Get
, use client.UnsafeDisableDeepCopy
option explicitly to aviod deepcopy if you won't modify returned objects.
It seems that inuse_space of eg1 and alloc_space of eg1 are the same picture
sorry, the link is mixed up, I will update it.
---update---
done
use client.UnsafeDisableDeepCopy option explicitly to aviod deepcopy if you won't modify returned objects.
According to your suggestion, I did following modification:
I updated above NewCache
function to:
NewCache: func(config *rest.Config, opts cache.Options) (cache.Cache, error) {
opts.DefaultTransform = fedinformer.StripUnusedFields
disable := true
opts.DefaultUnsafeDisableDeepCopy = &disable
return cache.New(config, opts)
},
As for case: 400 deployment + 400 secret + 800 configmap + 400 service + 400 serviceexport
before set DefaultUnsafeDisableDeepCopy
: 360MB RSS, once restart becomes 520MB.
after set DefaultUnsafeDisableDeepCopy
: 220MB RSS, once restart becomes 250MB.
before set
DefaultUnsafeDisableDeepCopy
: 360MB RSS, once restart becomes 520MB. after setDefaultUnsafeDisableDeepCopy
: 220MB RSS, once restart becomes 250MB.
@RainbowMango can we introduce this modification?
If use global configuration, we should check if there is case where returned objects by List
or Get
are modified in place.
What happened:
karmada-controller-manager has a high memory usage when it handles many resources.
scenario:
E.G1: 400 deployment + 400 secret + 800 configmap + 400 service + 400 serviceexport.
the RSS memory count to 360MB, besides, once restart karmada-controller-manager, the final RSS memory count to 520MB
E.G2: 2000 deployment + 2000 secret + 4000 configmap + 2000 service + 2000 serviceexport.
the RSS memory count to more than 1GB, besides, once restart karmada-controller-manager, the final RSS memory count to 2GB~3GB.
What problem do you want for hlep:
crictl stats
ortop
, the final memory is around 520MB, but if we observe inuse_space bypprof
, the inuse_space only shows 180M, where does the rest 340MB lies?How to reproduce it (as minimally and precisely as possible):
hack/local-up-karmada.sh
karmada-controller-manager
to 1:kubectl --context karmada-host -n karmada-system scale deploy karmada-controller-manager --replicas=1
cpp.yaml
、demo-test.yaml
、test.sh
in appendix into local file.test.sh
.Then, the
test.sh
will create 100 namespace, each namespace has 4 application, each application has 1 deployment/4 pod/1 secret/2 configmap/1 service/1 serviceexport.Finally, you can observe the RSS memory of karmada-controller-manager by
crictl stats
(ortop
) command, just like:and you can observe the heap memory collected by pprof, detail see: Profiling Karmada.
appendix:
cpp.yaml
```yaml apiVersion: policy.karmada.io/v1alpha1 kind: ClusterPropagationPolicy metadata: name: default-cpp spec: propagateDeps: true placement: replicaScheduling: replicaDivisionPreference: Weighted replicaSchedulingType: Divided weightPreference: staticWeightList: - targetCluster: clusterNames: - member1 weight: 1 - targetCluster: clusterNames: - member2 weight: 1 resourceSelectors: - apiVersion: apps/v1 kind: Deployment - apiVersion: v1 kind: Service - apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition name: serviceexports.multicluster.x-k8s.io - apiVersion: multicluster.x-k8s.io/v1alpha1 kind: ServiceExport - apiVersion: v1 kind: ConfigMap - apiVersion: v1 kind: Secret ```demo-test.yaml
```yaml apiVersion: apps/v1 kind: Deployment metadata: name: demo-test namespace: demo-test-ns spec: replicas: 4 selector: matchLabels: app: demo-test template: metadata: labels: app: demo-test spec: terminationGracePeriodSeconds: 0 containers: - image: nginx name: demo-test resources: limits: cpu: 10m memory: 10Mi volumes: - name: secret secret: secretName: demo-test - name: configmap configMap: name: demo-test - name: configmap-2 configMap: name: demo-test-2 --- apiVersion: v1 kind: Secret metadata: name: demo-test namespace: demo-test-ns type: Opaque data: key: MTIzCg== --- apiVersion: v1 kind: ConfigMap metadata: name: demo-test namespace: demo-test-ns data: test-key: "test-value" --- apiVersion: v1 kind: ConfigMap metadata: name: demo-test-2 namespace: demo-test-ns data: test-key: "test-value" --- apiVersion: v1 kind: Service metadata: name: demo-test namespace: demo-test-ns spec: ports: - port: 80 targetPort: 8080 selector: app: demo-test --- apiVersion: multicluster.x-k8s.io/v1alpha1 kind: ServiceExport metadata: name: demo-test namespace: demo-test-ns ```test.sh
```shell #!/bin/bash export KUBECONFIG=~/.kube/karmada.config:~/.kube/members.config alias ka='kubectl --context karmada-apiserver' shopt -s expand_aliases # ${ns} namespaces ns=100 # each namespace has ${dnum} deployment dnum=4 ka apply -f cpp.yaml # iterate each namespace for i in $(seq 1 $ns) do nsName="ns-${i}" # create namespace if it not exist ka create ns ${nsName} --dry-run=client -o yaml | ka apply -f - # create each resource in certain namespace for j in $(seq 1 $dnum) do cp -f demo-test.yaml demo-test-tmp.yaml sed -i'' -e "s/demo-test-ns/${nsName}/g" demo-test-tmp.yaml sed -i'' -e "s/demo-test-2/test-${i}-${j}-2/g" demo-test-tmp.yaml sed -i'' -e "s/demo-test/test-${i}-${j}/g" demo-test-tmp.yaml ka apply -f demo-test-tmp.yaml done done ```Anything else we need to know?:
pprof
:Environment:
kubectl-karmada version
orkarmadactl version
): latest