percona / percona-server-mongodb-operator

Percona Operator for MongoDB
https://www.percona.com/doc/kubernetes-operator-for-psmongodb/
Apache License 2.0
316 stars 138 forks source link

K8SPSMDB-1084 MongoDB connections leak in backup feature #1542

Closed steve-nzr closed 2 months ago

steve-nzr commented 2 months ago

K8SPSMDB-1084 Powered by Pull Request Badge

CHANGE DESCRIPTION

Problem:

We noticed an infinite increase, periodically, of opened connections count to our Mongod instances, on different clusters. It was really noticable on large sharded clusters (10 shards).

Cause:

Every backup, the operator left 1 connection open per ReplicaSet (per shard), in our case. So here for example : 11 connections per backup (10 shards + 1 ConfigServer).

Solution:

Disconnecting the client inside the psmb package after its use. I can't use defer since I'm in a loop, don't hesitate if you wan't to make it a new function instead.

Side note : This fix have been tested successfully in production since a few weeks now.

CHECKLIST

Jira

Tests

Config/Logging/Testability

CLAassistant commented 2 months ago

CLA assistant check
All committers have signed the CLA.

JNKPercona commented 2 months ago
Test name Status
arbiter passed
balancer passed
custom-replset-name passed
cross-site-sharded passed
data-at-rest-encryption passed
data-sharded passed
demand-backup passed
demand-backup-eks-credentials passed
demand-backup-physical passed
demand-backup-physical-sharded passed
demand-backup-sharded passed
expose-sharded passed
ignore-labels-annotations passed
init-deploy passed
finalizer passed
ldap passed
ldap-tls passed
limits passed
liveness passed
mongod-major-upgrade passed
mongod-major-upgrade-sharded passed
monitoring-2-0 passed
multi-cluster-service passed
non-voting passed
one-pod passed
operator-self-healing-chaos passed
pitr passed
pitr-sharded passed
pitr-physical passed
pvc-resize passed
recover-no-primary passed
rs-shard-migration passed
scaling passed
scheduled-backup passed
security-context passed
self-healing-chaos passed
service-per-pod passed
serviceless-external-nodes passed
smart-update passed
split-horizon passed
storage passed
tls-issue-cert-manager passed
upgrade passed
upgrade-consistency passed
upgrade-consistency-sharded-tls passed
upgrade-sharded passed
users passed
version-service passed
We run 48 out of 48

commit: https://github.com/percona/percona-server-mongodb-operator/pull/1542/commits/e268e038fea4db748efb581d0b5a8cd7a5ca4aef image: perconalab/percona-server-mongodb-operator:PR-1542-e268e038

hors commented 2 months ago

@steve-nzr Thank you for your contribution. We will include it in the next release.