"Failed to get $dbstats for database XXX: canceled while checking out a connection from connection pool: context canceled; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0" "Failed to get $dbstats for database XXX: context deadline exceeded"

Hello, let me first say sorry. I am using Google Translate to ask questions. The description of the question may not be very clear. Please understand. My English is not that good.

Describe the bug When I use the following mongodb-exporter-deployment to connect two other identical databases with different data volumes, no error will be reported if the data volume is small. I have created several Databases and Collections and they can be collected normally. However, when the amount of data is large, some indicators cannot be obtained, and an error is reported in the log: "Failed to get $dbstats for database worksheetdb: canceled while checking out a connection from connection pool: context canceled; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0" "Failed to get $dbstats for database worksheetdb: context deadline exceeded" To Reproduce Steps to reproduce the behavior:

describe steps to reproduce the issue

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongodb-exporter
namespace: monitoring
spec:
selector:
matchLabels:
  app: mongodb-exporter
replicas: 1
template:
metadata:
  labels:
    app: mongodb-exporter
  annotations:
    prometheus.io/port: "9216"
    prometheus.io/scrape: "true"
spec:
  containers:
    - name: mongodb-exporter
      imagePullPolicy: IfNotPresent
      image: percona/mongodb_exporter:0.40.0
      env:
        - name: MONGODB_URI
          valueFrom:
            secretKeyRef:
              name: mongodb-secret-test
              key: datasource
      args:
        - "--discovering-mode"
        - "--compatible-mode"
        - "--collect-all"
        - '--mongodb.collstats-colls=config.App,config.ApplicationOverride,config.Caller,config.CommonRules,config.HabilitationOverride,config.IconSvg,config.UpdateStatusEntity,config.feature'
        # - "--log.level=debug"
        - "--mongodb.connect-timeout-ms=30000"
      ports:
        - containerPort: 9216
          name: metric-port
      resources:
        limits:
          memory: "256Mi"
          cpu: "256m"
---
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: "9216"
name: mongodb-exporter-service
namespace: monitoring
spec:
selector:
app: mongodb-exporter
type: ClusterIP
ports:
- name: metric-port
  port: 9216
  protocol: TCP
  targetPort: 9216
  # nodePort: 31082
---
apiVersion: v1
kind: Secret
metadata:
name: mongodb-secret-test
namespace: monitoring
type: Opaque
stringData:
datasource: "mongodb://USER:PASSWORD@{host1}:27017/admin"

Expected behavior The log does not report errors and all indicators can be collected normally.

Logs I have omitted some duplicate logs to improve readability

level=info ts=2024-02-04T03:43:06.832Z caller=tls_config.go:274 msg="Listening on" address=[::]:9216
level=info ts=2024-02-04T03:43:06.833Z caller=tls_config.go:277 msg="TLS is disabled." http2=false address=[::]:9216
time="2024-02-04T03:43:15Z" level=error msg="cannot get $indexStats cursor for collection modeldb.system.profile: (Unauthorized) not authorized on modeldb to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"01b14739-f158-4aff-9d16-4fd9aa3ad3e8\") }, $db: \"modeldb\" }"
time="2024-02-04T03:43:16Z" level=error msg="cannot get $indexStats cursor for collection basedb.system.profile: (Unauthorized) not authorized on basedb to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"01b14739-f158-4aff-9d16-4fd9aa3ad3e8\") }, $db: \"basedb\" }"
time="2024-02-04T03:43:17Z" level=error msg="cannot get $indexStats cursor for collection formcenter.system.profile: (Unauthorized) not authorized on formcenter to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"01b14739-f158-4aff-9d16-4fd9aa3ad3e8\") }, $db: \"formcenter\" }"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.scenes: connection(192.168.1.145:37017[-13]) incomplete read of message header: context deadline exceeded"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.settings: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.taskentities: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.taskpackages: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.volumeweights: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection csidb.accesssettings: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
···
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection airdb.weathers: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection airdb.winds: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.branches: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.branchtemplates: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.branchtypes: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.branchtypetemplates: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.classifications: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.classificationtemplates: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.inspectionlots: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.linkmodels: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection operatingdb.mqcollections: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection yzstore_private_serverdb.appkeys: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection yzstore_private_serverdb.mqcollections: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection yzstore_private_serverdb.template_buys: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection yzstore_private_serverdb.template_uploads: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.archivesettings: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.departmentgroups: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.drawingtypes: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.drawingunits: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.exportrecords: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
···
time="2024-02-04T03:43:20Z" level=error msg="cannot get $indexStats cursor for collection thirdpartyauthdb.system.profile: (Unauthorized) not authorized on thirdpartyauthdb to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"01d9a45c-b6ea-444d-a9d5-14d8ce78130e\") }, $db: \"thirdpartyauthdb\" }"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection formcenter.system.profile: (Unauthorized) not authorized on formcenter to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"f5d21336-0a4c-45b7-92ce-8dd540f7c5a9\") }, $db: \"formcenter\" }"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection workflowdb.system.profile: (Unauthorized) not authorized on workflowdb to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"f5d21336-0a4c-45b7-92ce-8dd540f7c5a9\") }, $db: \"workflowdb\" }"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection mixindb.mqcollections: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.customformdatas: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.customtemplatedatas: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.customtemplatefields: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.forms: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.fromtemplates: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.mqcollections: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.sheetformdatas: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.sheettemplatedatas: context deadline exceeded"
···

Environment

Centos 7.9 uname 3.10.0-1160.62.1.el7.x86_64 ,
k8s v1.24.10
MongoDB version v4.0.28

Additional context Add any other context about the problem here. I also tried to collect the database separately through the --collector.dbstats parameter. There was no error in the execution, but there were also database collection errors in the logs collected through the --collector.all parameter.

percona / mongodb_exporter