TiDB Operator OOM - Githubissues

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.

https://docs.pingcap.com/tidb-in-kubernetes/

Apache License 2.0

1.24k stars 499 forks source link

TiDB Operator OOM #3082

Open DanielZhangQD opened 4 years ago

DanielZhangQD commented 4 years ago

Bug Report

What version of Kubernetes are you using?

v1.15.3 What version of TiDB Operator are you using?

v1.1.2 What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

What's the status of the TiDB cluster pods?

What did you do?

What did you expect to see? TiDB Operator works normally without OOM killed What did you see instead? TiDB Operator takes several GB memory and eventually gets OOM killed

DanielZhangQD commented 4 years ago

Some info:

TLS is not enabled
TiDB Cluster shares the Kubernetes cluster with many other applications

Memory usage:

Describe the Pod:

Heap profile:

weekface commented 4 years ago

This is heap profile:

tidb-controller-manager.pb.gz

This cluster has too many pods. So this must be caused by podInformer cache.

The solution: make a new informerFactory for podInformer, and sets a custom filter on all listers of the configured SharedInformerFactory.

kubeinformers.WithTweakListOptions(func(corev1.ListOptions))

@cofyc @DanielZhangQD PTAL