search

kyma-project / module-manager

Apache License 2.0
3 stars 13 forks source link

High memory consumption of module-manager #141

Closed ruanxin closed 1 year ago

ruanxin commented 1 year ago

Deployed 100 Kyma with 2000 manifest CR, the peak memory consumption reach 14GB (memory limit configured 16GB) module-manager

pprof insights

(pprof) top
Showing nodes accounting for 3.75GB, 88.11% of 4.25GB total
Dropped 409 nodes (cum <= 0.02GB)
Showing top 10 nodes out of 138
      flat  flat%   sum%        cum   cum%
    1.96GB 46.15% 46.15%     1.96GB 46.15%  reflect.New
    0.84GB 19.67% 65.82%     0.84GB 19.67%  google.golang.org/protobuf/internal/impl.consumeStringValidateUTF8
    0.35GB  8.17% 73.99%     0.35GB  8.25%  io.ReadAll
    0.19GB  4.56% 78.55%     0.19GB  4.56%  k8s.io/apimachinery/pkg/api/meta.(*DefaultRESTMapper).AddSpecific
    0.13GB  2.96% 81.52%     0.36GB  8.54%  k8s.io/kube-openapi/pkg/util/proto.(*Definitions).parseKind
    0.08GB  1.94% 83.46%     0.08GB  1.94%  google.golang.org/protobuf/internal/impl.consumeStringSliceValidateUTF8
    0.06GB  1.39% 84.85%     0.15GB  3.49%  k8s.io/kube-openapi/pkg/util/proto.VendorExtensionToMap
    0.05GB  1.28% 86.13%     0.05GB  1.28%  gopkg.in/yaml%2ev2.yaml_parser_initialize (inline)
    0.04GB     1% 87.12%     0.04GB     1%  google.golang.org/protobuf/internal/impl.pointer.AppendPointerSlice
    0.04GB  0.99% 88.11%     0.07GB  1.65%  k8s.io/kube-openapi/pkg/util/proto.(*Definitions).parsePrimitive

heap profile: 4435: 354760696 [210034: 40235312400] @ heap/1048576
12: 51806208 [1019: 4399210496] @ 0x4b0185 0x99bab4 0x99ba99 0x99b645 0x99b0fe 0x99b0df 0x99ae33 0x99b4a9 0x1247a58 0x17c8f62 0x17829ea 0x484ae2 0x178298a 0x1782955 0x1782b0e 0x484ae2 0x1782aaa 0x1782a75 0x17b30b3 0x17b2e9e 0x17bc222 0x195e03b 0x195f4e9 0x195fba5 0x195efb8 0x1968bcc 0x196b435 0x46df61
#   0x4b0184    io.ReadAll+0xa4                                                         /usr/local/go/src/io/io.go:659
#   0x99bab3    io/ioutil.ReadAll+0xb3                                                      /usr/local/go/src/io/ioutil/ioutil.go:27
#   0x99ba98    k8s.io/client-go/rest.(*Request).transformResponse+0x98                                     /go/pkg/mod/k8s.io/client-go@v0.25.2/rest/request.go:1040
#   0x99b644    k8s.io/client-go/rest.(*Request).Do.func1+0x44                                          /go/pkg/mod/k8s.io/client-go@v0.25.2/rest/request.go:1006
#   0x99b0fd    k8s.io/client-go/rest.(*Request).request.func3.1+0xdd                                       /go/pkg/mod/k8s.io/client-go@v0.25.2/rest/request.go:981
#   0x99b0de    k8s.io/client-go/rest.(*Request).request.func3+0xbe                                     /go/pkg/mod/k8s.io/client-go@v0.25.2/rest/request.go:988
#   0x99ae32    k8s.io/client-go/rest.(*Request).request+0x4f2                                          /go/pkg/mod/k8s.io/client-go@v0.25.2/rest/request.go:990
#   0x99b4a8    k8s.io/client-go/rest.(*Request).Do+0xc8                                            /go/pkg/mod/k8s.io/client-go@v0.25.2/rest/request.go:1005
#   0x1247a57   k8s.io/client-go/discovery.(*DiscoveryClient).OpenAPISchema+0xd7                                /go/pkg/mod/k8s.io/client-go@v0.25.2/discovery/discovery_client.go:414
#   0x17c8f61   k8s.io/client-go/discovery/cached/memory.(*memCacheClient).OpenAPISchema+0x21                           /go/pkg/mod/k8s.io/client-go@v0.25.2/discovery/cached/memory/memcache.go:146
#   0x17829e9   k8s.io/kubectl/pkg/util/openapi.(*CachedOpenAPIGetter).OpenAPISchema.func1+0x29                         /go/pkg/mod/k8s.io/kubectl@v0.25.2/pkg/util/openapi/openapi_getter.go:49
#   0x484ae1    sync.(*Once).doSlow+0xc1                                                    /usr/local/go/src/sync/once.go:74
#   0x1782989   sync.(*Once).Do+0x49                                                        /usr/local/go/src/sync/once.go:65
#   0x1782954   k8s.io/kubectl/pkg/util/openapi.(*CachedOpenAPIGetter).OpenAPISchema+0x14                           /go/pkg/mod/k8s.io/kubectl@v0.25.2/pkg/util/openapi/openapi_getter.go:48
#   0x1782b0d   k8s.io/kubectl/pkg/util/openapi.(*CachedOpenAPIParser).Parse.func1+0x2d                             /go/pkg/mod/k8s.io/kubectl@v0.25.2/pkg/util/openapi/openapi_getter.go:73
#   0x484ae1    sync.(*Once).doSlow+0xc1                                                    /usr/local/go/src/sync/once.go:74
#   0x1782aa9   sync.(*Once).Do+0x49                                                        /usr/local/go/src/sync/once.go:65
#   0x1782a74   k8s.io/kubectl/pkg/util/openapi.(*CachedOpenAPIParser).Parse+0x14                               /go/pkg/mod/k8s.io/kubectl@v0.25.2/pkg/util/openapi/openapi_getter.go:72
#   0x17b30b2   k8s.io/kubectl/pkg/cmd/util.(*factoryImpl).OpenAPISchema+0x72                                   /go/pkg/mod/k8s.io/kubectl@v0.25.2/pkg/cmd/util/factory_client_access.go:183
#   0x17b2e9d   k8s.io/kubectl/pkg/cmd/util.(*factoryImpl).Validator+0x5d                                   /go/pkg/mod/k8s.io/kubectl@v0.25.2/pkg/cmd/util/factory_client_access.go:156
#   0x17bc221   helm.sh/helm/v3/pkg/kube.(*Client).Build+0x1c1                                          /go/pkg/mod/helm.sh/helm/v3@v3.10.1/pkg/kube/client.go:206
#   0x195e03a   github.com/kyma-project/module-manager/operator/pkg/manifest.(*HelmClient).GetTargetResources+0xfa              /workspace/pkg/manifest/client.go:302
#   0x195f4e8   github.com/kyma-project/module-manager/operator/pkg/manifest.(*Operations).getClusterResources+0x1a8                /workspace/pkg/manifest/operations.go:181
#   0x195fba4   github.com/kyma-project/module-manager/operator/pkg/manifest.(*Operations).consistencyCheck+0x1a4               /workspace/pkg/manifest/operations.go:220
#   0x195efb7   github.com/kyma-project/module-manager/operator/pkg/manifest.ConsistencyCheck+0xb7                      /workspace/pkg/manifest/operations.go:133
#   0x1968bcb   github.com/kyma-project/module-manager/operator/controllers.(*ManifestReconciler).checkInstallInfo+0x1eb            /workspace/controllers/manifest_controller.go:230
#   0x196b434   github.com/kyma-project/module-manager/operator/controllers.(*ManifestWorkerPool).StartConsistencyCheckWorkers.func1+0x174  /workspace/controllers/workers.go:66

pprof caused by helm build: https://github.com/fluxcd/helm-controller/issues/345 root cause might be: https://github.com/kubernetes/kubernetes/issues/105932

goroutine stack dump

goroutine stack dump when no manifest deployed goroutine stack dump during 2000 manifest deploying

jakobmoellerdev commented 1 year ago

I think this can be (partially) circumvented by not calling helm build based on the cached repository, but by parsing the CRDs into Manifests and then applying those. I assume this is where our resource spike is coming from.