mongodb-agent never gets ready

[kennedyoliveira]

What did you do to encounter the bug? Steps to reproduce the behavior:

1. Installed the operator in the mongodb namespace
2. Deployed a mongodb replica-set with minimal configuraiton

What did you expect? Expected the instances to start and become ready

What happened instead? First instance of the replica set never gets ready and keep restarting.

Screenshots If applicable, add screenshots to help explain your problem.

Operator Information

• Operator Version: quay.io/mongodb/mongodb-kubernetes-operator:0.9.0
• MongoDB Image used: docker.io/mongo:7.0.7-jammy

Kubernetes Cluster Information

• Distribution: k3s
• Version: 1.28.5
• Image Registry location (quay, or an internal registry): quay

Additional context Add any other context about the problem here.

Running on a arm64 sbc cluster.

If possible, please include:

• The operator logs
• Below we assume that your replicaset database pods are named mongo-<>. For instance:

❯ k get pods
NAME                                           READY   STATUS             RESTARTS      AGE
example-mongodb-0                              0/2     CrashLoopBackOff   8 (85s ago)   18m
mongodb-kubernetes-operator-58f56cd55d-9gndv   1/1     Running            0             10h

❯ k get mdbc
NAME              PHASE     VERSION
example-mongodb   Pending

• yaml definitions of your MongoDB Deployment(s):
  • kubectl get mdbc -oyaml

apiVersion: v1
items:
- apiVersion: mongodbcommunity.mongodb.com/v1
  kind: MongoDBCommunity
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"mongodbcommunity.mongodb.com/v1","kind":"MongoDBCommunity","metadata":{"annotations":{},"name":"example-mongodb","namespace":"mongodb"},"spec":{"additionalMongodConfig":{"storage.wiredTiger.engineConfig.journalCompressor":"zlib"},"members":3,"security":{"authentication":{"modes":["SCRAM"]}},"statefulSet":{"spec":{"template":{"spec":{"containers":[{"name":"mongodb-agent","readinessProbe":{"initialDelaySeconds":30,"periodSeconds":20,"timeoutSeconds":20}}],"nodeSelector":{"workload/io":"true"}}}}},"type":"ReplicaSet","users":[{"db":"admin","name":"my-user","passwordSecretRef":{"name":"my-user-password"},"roles":[{"db":"admin","name":"clusterAdmin"},{"db":"admin","name":"userAdminAnyDatabase"}],"scramCredentialsSecretName":"my-scram"}],"version":"7.0.7-jammy"}}
    creationTimestamp: "2024-03-29T05:01:34Z"
    generation: 1
    name: example-mongodb
    namespace: mongodb
    resourceVersion: "279642631"
    uid: f634142a-d2cb-4e02-a6a5-27fefde47457
  spec:
    additionalMongodConfig:
      storage.wiredTiger.engineConfig.journalCompressor: zlib
    members: 3
    security:
      authentication:
        ignoreUnknownUsers: true
        modes:
        - SCRAM
    statefulSet:
      spec:
        template:
          spec:
            containers:
            - name: mongodb-agent
              readinessProbe:
                initialDelaySeconds: 30
                periodSeconds: 20
                timeoutSeconds: 20
            nodeSelector:
              workload/io: "true"
    type: ReplicaSet
    users:
    - db: admin
      name: my-user
      passwordSecretRef:
        name: my-user-password
      roles:
      - db: admin
        name: clusterAdmin
      - db: admin
        name: userAdminAnyDatabase
      scramCredentialsSecretName: my-scram
    version: 7.0.7-jammy
  status:
    currentMongoDBMembers: 0
    currentStatefulSetReplicas: 0
    message: ReplicaSet is not yet ready, retrying in 10 seconds
    mongoUri: ""
    phase: Pending
kind: List
metadata:
  resourceVersion: ""

• yaml definitions of your kubernetes objects like the statefulset(s), pods (we need to see the state of the containers):
  • kubectl get sts -oyaml
  • kubectl get pods -oyaml

apiVersion: v1
items:
- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    creationTimestamp: "2024-03-29T05:01:34Z"
    generation: 1
    name: example-mongodb
    namespace: mongodb
    ownerReferences:
    - apiVersion: mongodbcommunity.mongodb.com/v1
      blockOwnerDeletion: true
      controller: true
      kind: MongoDBCommunity
      name: example-mongodb
      uid: f634142a-d2cb-4e02-a6a5-27fefde47457
    resourceVersion: "279843431"
    uid: f7e37b9a-552e-4b27-95d5-9d414ed32bec
  spec:
    persistentVolumeClaimRetentionPolicy:
      whenDeleted: Retain
      whenScaled: Retain
    podManagementPolicy: OrderedReady
    replicas: 3
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: example-mongodb-svc
    serviceName: example-mongodb-svc
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: example-mongodb-svc
      spec:
        containers:
        - args:
          - ""
          command:
          - /bin/sh
          - -c
          - |2+

            #run post-start hook to handle version changes
            /hooks/version-upgrade

            # wait for config and keyfile to be created by the agent
             while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

            # start mongod with this configuration
            exec mongod -f /data/automation-mongod.conf;

          env:
          - name: AGENT_STATUS_FILEPATH
            value: /healthstatus/agent-health-status.json
          image: docker.io/mongo:7.0.7-jammy
          imagePullPolicy: IfNotPresent
          name: mongod
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /data
            name: data-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: example-mongodb-keyfile
          - mountPath: /healthstatus
            name: healthstatus
          - mountPath: /hooks
            name: hooks
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /tmp
            name: tmp
        - command:
          - /bin/bash
          - -c
          - |-
            current_uid=$(id -u)
            AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
            declare -r current_uid
            if ! grep -q "${current_uid}" /etc/passwd ; then
            sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
            echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
            export NSS_WRAPPER_PASSWD=/tmp/passwd
            export LD_PRELOAD=libnss_wrapper.so
            export NSS_WRAPPER_GROUP=/etc/group
            fi
            agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile ${AGENT_LOG_FILE} -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS} -logLevel ${AGENT_LOG_LEVEL}
          env:
          - name: AGENT_LOG_FILE
            value: /var/log/mongodb-mms-automation/automation-agent.log
          - name: AGENT_LOG_LEVEL
            value: INFO
          - name: AGENT_MAX_LOG_FILE_DURATION_HOURS
            value: "24"
          - name: AGENT_STATUS_FILEPATH
            value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
          - name: AUTOMATION_CONFIG_MAP
            value: example-mongodb-config
          - name: HEADLESS_AGENT
            value: "true"
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          image: quay.io/mongodb/mongodb-agent-ubi:107.0.0.8465-1-arm64
          imagePullPolicy: Always
          name: mongodb-agent
          readinessProbe:
            exec:
              command:
              - /opt/scripts/readinessprobe
            failureThreshold: 40
            initialDelaySeconds: 30
            periodSeconds: 20
            successThreshold: 1
            timeoutSeconds: 20
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
          - mountPath: /var/lib/automation/config
            name: automation-config
            readOnly: true
          - mountPath: /data
            name: data-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: example-mongodb-keyfile
          - mountPath: /var/log/mongodb-mms-automation/healthstatus
            name: healthstatus
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /tmp
            name: tmp
        dnsPolicy: ClusterFirst
        initContainers:
        - command:
          - cp
          - version-upgrade-hook
          - /hooks/version-upgrade
          image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
          imagePullPolicy: Always
          name: mongod-posthook
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /hooks
            name: hooks
        - command:
          - cp
          - /probes/readinessprobe
          - /opt/scripts/readinessprobe
          image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
          imagePullPolicy: Always
          name: mongodb-agent-readinessprobe
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
        nodeSelector:
          workload/io: "true"
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 2000
          runAsNonRoot: true
          runAsUser: 2000
        serviceAccount: mongodb-database
        serviceAccountName: mongodb-database
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: agent-scripts
        - name: automation-config
          secret:
            defaultMode: 416
            secretName: example-mongodb-config
        - emptyDir: {}
          name: example-mongodb-keyfile
        - emptyDir: {}
          name: healthstatus
        - emptyDir: {}
          name: hooks
        - emptyDir: {}
          name: tmp
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10G
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 2G
        volumeMode: Filesystem
      status:
        phase: Pending
  status:
    availableReplicas: 0
    collisionCount: 0
    currentReplicas: 1
    currentRevision: example-mongodb-6589fbc778
    observedGeneration: 1
    replicas: 1
    updateRevision: example-mongodb-6589fbc778
    updatedReplicas: 1
- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    creationTimestamp: "2024-03-29T05:01:34Z"
    generation: 1
    name: example-mongodb-arb
    namespace: mongodb
    ownerReferences:
    - apiVersion: mongodbcommunity.mongodb.com/v1
      blockOwnerDeletion: true
      controller: true
      kind: MongoDBCommunity
      name: example-mongodb
      uid: f634142a-d2cb-4e02-a6a5-27fefde47457
    resourceVersion: "279642633"
    uid: b2384db6-92e4-4746-a3df-d8b9c440f890
  spec:
    persistentVolumeClaimRetentionPolicy:
      whenDeleted: Retain
      whenScaled: Retain
    podManagementPolicy: OrderedReady
    replicas: 0
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: example-mongodb-svc
    serviceName: example-mongodb-svc
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: example-mongodb-svc
      spec:
        containers:
        - args:
          - ""
          command:
          - /bin/sh
          - -c
          - |2+

            #run post-start hook to handle version changes
            /hooks/version-upgrade

            # wait for config and keyfile to be created by the agent
             while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

            # start mongod with this configuration
            exec mongod -f /data/automation-mongod.conf;

          env:
          - name: AGENT_STATUS_FILEPATH
            value: /healthstatus/agent-health-status.json
          image: docker.io/mongo:7.0.7-jammy
          imagePullPolicy: IfNotPresent
          name: mongod
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /data
            name: data-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: example-mongodb-keyfile
          - mountPath: /healthstatus
            name: healthstatus
          - mountPath: /hooks
            name: hooks
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /tmp
            name: tmp
        - command:
          - /bin/bash
          - -c
          - |-
            current_uid=$(id -u)
            AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
            declare -r current_uid
            if ! grep -q "${current_uid}" /etc/passwd ; then
            sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
            echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
            export NSS_WRAPPER_PASSWD=/tmp/passwd
            export LD_PRELOAD=libnss_wrapper.so
            export NSS_WRAPPER_GROUP=/etc/group
            fi
            agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile ${AGENT_LOG_FILE} -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS} -logLevel ${AGENT_LOG_LEVEL}
          env:
          - name: AGENT_LOG_FILE
            value: /var/log/mongodb-mms-automation/automation-agent.log
          - name: AGENT_LOG_LEVEL
            value: INFO
          - name: AGENT_MAX_LOG_FILE_DURATION_HOURS
            value: "24"
          - name: AGENT_STATUS_FILEPATH
            value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
          - name: AUTOMATION_CONFIG_MAP
            value: example-mongodb-config
          - name: HEADLESS_AGENT
            value: "true"
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          image: quay.io/mongodb/mongodb-agent-ubi:107.0.0.8465-1-arm64
          imagePullPolicy: Always
          name: mongodb-agent
          readinessProbe:
            exec:
              command:
              - /opt/scripts/readinessprobe
            failureThreshold: 40
            initialDelaySeconds: 30
            periodSeconds: 20
            successThreshold: 1
            timeoutSeconds: 20
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
          - mountPath: /var/lib/automation/config
            name: automation-config
            readOnly: true
          - mountPath: /data
            name: data-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: example-mongodb-keyfile
          - mountPath: /var/log/mongodb-mms-automation/healthstatus
            name: healthstatus
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /tmp
            name: tmp
        dnsPolicy: ClusterFirst
        initContainers:
        - command:
          - cp
          - version-upgrade-hook
          - /hooks/version-upgrade
          image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
          imagePullPolicy: Always
          name: mongod-posthook
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /hooks
            name: hooks
        - command:
          - cp
          - /probes/readinessprobe
          - /opt/scripts/readinessprobe
          image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
          imagePullPolicy: Always
          name: mongodb-agent-readinessprobe
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
        nodeSelector:
          workload/io: "true"
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 2000
          runAsNonRoot: true
          runAsUser: 2000
        serviceAccount: mongodb-database
        serviceAccountName: mongodb-database
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: agent-scripts
        - name: automation-config
          secret:
            defaultMode: 416
            secretName: example-mongodb-config
        - emptyDir: {}
          name: example-mongodb-keyfile
        - emptyDir: {}
          name: healthstatus
        - emptyDir: {}
          name: hooks
        - emptyDir: {}
          name: tmp
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10G
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 2G
        volumeMode: Filesystem
      status:
        phase: Pending
  status:
    availableReplicas: 0
    collisionCount: 0
    currentRevision: example-mongodb-arb-6589fbc778
    observedGeneration: 1
    replicas: 0
    updateRevision: example-mongodb-arb-6589fbc778
kind: List
metadata:
  resourceVersion: ""

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      agent.mongodb.com/version: "-1"
    creationTimestamp: "2024-03-29T14:33:57Z"
    generateName: example-mongodb-
    labels:
      app: example-mongodb-svc
      apps.kubernetes.io/pod-index: "0"
      controller-revision-hash: example-mongodb-6589fbc778
      statefulset.kubernetes.io/pod-name: example-mongodb-0
    name: example-mongodb-0
    namespace: mongodb
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: StatefulSet
      name: example-mongodb
      uid: f7e37b9a-552e-4b27-95d5-9d414ed32bec
    resourceVersion: "279849453"
    uid: cdcfee1c-cf93-4f80-95ba-656910870900
  spec:
    containers:
    - args:
      - ""
      command:
      - /bin/sh
      - -c
      - |2+

        #run post-start hook to handle version changes
        /hooks/version-upgrade

        # wait for config and keyfile to be created by the agent
         while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

        # start mongod with this configuration
        exec mongod -f /data/automation-mongod.conf;

      env:
      - name: AGENT_STATUS_FILEPATH
        value: /healthstatus/agent-health-status.json
      image: docker.io/mongo:7.0.7-jammy
      imagePullPolicy: IfNotPresent
      name: mongod
      resources:
        limits:
          cpu: "1"
          memory: 500M
        requests:
          cpu: 500m
          memory: 400M
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /data
        name: data-volume
      - mountPath: /var/lib/mongodb-mms-automation/authentication
        name: example-mongodb-keyfile
      - mountPath: /healthstatus
        name: healthstatus
      - mountPath: /hooks
        name: hooks
      - mountPath: /var/log/mongodb-mms-automation
        name: logs-volume
      - mountPath: /tmp
        name: tmp
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-rmpss
        readOnly: true
    - command:
      - /bin/bash
      - -c
      - |-
        current_uid=$(id -u)
        AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
        declare -r current_uid
        if ! grep -q "${current_uid}" /etc/passwd ; then
        sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
        echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
        export NSS_WRAPPER_PASSWD=/tmp/passwd
        export LD_PRELOAD=libnss_wrapper.so
        export NSS_WRAPPER_GROUP=/etc/group
        fi
        agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile ${AGENT_LOG_FILE} -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS} -logLevel ${AGENT_LOG_LEVEL}
      env:
      - name: AGENT_LOG_FILE
        value: /var/log/mongodb-mms-automation/automation-agent.log
      - name: AGENT_LOG_LEVEL
        value: INFO
      - name: AGENT_MAX_LOG_FILE_DURATION_HOURS
        value: "24"
      - name: AGENT_STATUS_FILEPATH
        value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
      - name: AUTOMATION_CONFIG_MAP
        value: example-mongodb-config
      - name: HEADLESS_AGENT
        value: "true"
      - name: POD_NAMESPACE
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
      image: quay.io/mongodb/mongodb-agent-ubi:107.0.0.8465-1-arm64
      imagePullPolicy: Always
      name: mongodb-agent
      readinessProbe:
        exec:
          command:
          - /opt/scripts/readinessprobe
        failureThreshold: 40
        initialDelaySeconds: 30
        periodSeconds: 20
        successThreshold: 1
        timeoutSeconds: 20
      resources:
        limits:
          cpu: "1"
          memory: 500M
        requests:
          cpu: 500m
          memory: 400M
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /opt/scripts
        name: agent-scripts
      - mountPath: /var/lib/automation/config
        name: automation-config
        readOnly: true
      - mountPath: /data
        name: data-volume
      - mountPath: /var/lib/mongodb-mms-automation/authentication
        name: example-mongodb-keyfile
      - mountPath: /var/log/mongodb-mms-automation/healthstatus
        name: healthstatus
      - mountPath: /var/log/mongodb-mms-automation
        name: logs-volume
      - mountPath: /tmp
        name: tmp
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-rmpss
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    hostname: example-mongodb-0
    initContainers:
    - command:
      - cp
      - version-upgrade-hook
      - /hooks/version-upgrade
      image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
      imagePullPolicy: Always
      name: mongod-posthook
      resources: {}
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /hooks
        name: hooks
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-rmpss
        readOnly: true
    - command:
      - cp
      - /probes/readinessprobe
      - /opt/scripts/readinessprobe
      image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
      imagePullPolicy: Always
      name: mongodb-agent-readinessprobe
      resources: {}
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /opt/scripts
        name: agent-scripts
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-rmpss
        readOnly: true
    nodeName: k3s-worker-4-rock-4aplus
    nodeSelector:
      workload/io: "true"
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
      fsGroup: 2000
      runAsNonRoot: true
      runAsUser: 2000
    serviceAccount: mongodb-database
    serviceAccountName: mongodb-database
    subdomain: example-mongodb-svc
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: data-volume
      persistentVolumeClaim:
        claimName: data-volume-example-mongodb-0
    - name: logs-volume
      persistentVolumeClaim:
        claimName: logs-volume-example-mongodb-0
    - emptyDir: {}
      name: agent-scripts
    - name: automation-config
      secret:
        defaultMode: 416
        secretName: example-mongodb-config
    - emptyDir: {}
      name: example-mongodb-keyfile
    - emptyDir: {}
      name: healthstatus
    - emptyDir: {}
      name: hooks
    - emptyDir: {}
      name: tmp
    - name: kube-api-access-rmpss
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2024-03-29T14:34:17Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2024-03-29T14:33:57Z"
      message: 'containers with unready status: [mongod mongodb-agent]'
      reason: ContainersNotReady
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2024-03-29T14:33:57Z"
      message: 'containers with unready status: [mongod mongodb-agent]'
      reason: ContainersNotReady
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2024-03-29T14:33:57Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://be1f0e8697c08981d2495b3a39d3bb1b9d11794d51136b0afde62ea922495bb0
      image: docker.io/library/mongo:7.0.7-jammy
      imageID: docker.io/library/mongo@sha256:0c929d1cfb407ee7b5d421e0acc2bb4283ba0555f940ccd4860085d9de3a7e97
      lastState:
        terminated:
          containerID: containerd://be1f0e8697c08981d2495b3a39d3bb1b9d11794d51136b0afde62ea922495bb0
          exitCode: 132
          finishedAt: "2024-03-29T14:50:55Z"
          reason: Error
          startedAt: "2024-03-29T14:50:52Z"
      name: mongod
      ready: false
      restartCount: 8
      started: false
      state:
        waiting:
          message: back-off 5m0s restarting failed container=mongod pod=example-mongodb-0_mongodb(cdcfee1c-cf93-4f80-95ba-656910870900)
          reason: CrashLoopBackOff
    - containerID: containerd://cf011a75de6568b7a38d34e822bbd07bb2cd176182dccebcdff738526d8cc789
      image: quay.io/mongodb/mongodb-agent-ubi:107.0.0.8465-1-arm64
      imageID: quay.io/mongodb/mongodb-agent-ubi@sha256:a83189c15aebeaee1ea47bddb87cb0562b69ab208c7ce23a8aa8b68e8f185564
      lastState: {}
      name: mongodb-agent
      ready: false
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2024-03-29T14:34:40Z"
    hostIP: 192.168.50.33
    initContainerStatuses:
    - containerID: containerd://92094b299f1c2462715eb9e5dd4bf4a63383652a426f97bded8242940e91c9d7
      image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
      imageID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:c2c01822a389f57b6c87404a1097765acecb9e6ac0f297c123e4e134f93fc6af
      lastState: {}
      name: mongod-posthook
      ready: true
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: containerd://92094b299f1c2462715eb9e5dd4bf4a63383652a426f97bded8242940e91c9d7
          exitCode: 0
          finishedAt: "2024-03-29T14:34:12Z"
          reason: Completed
          startedAt: "2024-03-29T14:34:12Z"
    - containerID: containerd://f2f2aec536b6d6f87a2fd5ab2e9538c262239ca726a971523c44c55962ccd168
      image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
      imageID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:81704c64bde4e61836af17f7b14bd009034a68d39c2399b1fad9eb1c010d0a22
      lastState: {}
      name: mongodb-agent-readinessprobe
      ready: true
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: containerd://f2f2aec536b6d6f87a2fd5ab2e9538c262239ca726a971523c44c55962ccd168
          exitCode: 0
          finishedAt: "2024-03-29T14:34:15Z"
          reason: Completed
          startedAt: "2024-03-29T14:34:15Z"
    phase: Running
    podIP: 10.42.5.30
    podIPs:
    - ip: 10.42.5.30
    qosClass: Burstable
    startTime: "2024-03-29T14:33:57Z"
- apiVersion: v1
  kind: Pod
  metadata:
    creationTimestamp: "2024-03-29T04:06:31Z"
    generateName: mongodb-kubernetes-operator-58f56cd55d-
    labels:
      name: mongodb-kubernetes-operator
      pod-template-hash: 58f56cd55d
    name: mongodb-kubernetes-operator-58f56cd55d-9gndv
    namespace: mongodb
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: mongodb-kubernetes-operator-58f56cd55d
      uid: 652b7bea-ccf4-4734-9291-113ea4d33262
    resourceVersion: "279621345"
    uid: bd31b0c8-8884-47f5-8ba0-57e3cceb16e4
  spec:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: name
              operator: In
              values:
              - mongodb-kubernetes-operator
          topologyKey: kubernetes.io/hostname
    containers:
    - command:
      - /usr/local/bin/entrypoint
      env:
      - name: WATCH_NAMESPACE
        value: '*'
      - name: POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: OPERATOR_NAME
        value: mongodb-kubernetes-operator
      - name: AGENT_IMAGE
        value: quay.io/mongodb/mongodb-agent-ubi:107.0.0.8465-1-arm64
      - name: VERSION_UPGRADE_HOOK_IMAGE
        value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
      - name: READINESS_PROBE_IMAGE
        value: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
      - name: MONGODB_IMAGE
        value: mongo
      - name: MONGODB_REPO_URL
        value: docker.io
      image: quay.io/mongodb/mongodb-kubernetes-operator:0.9.0
      imagePullPolicy: Always
      name: mongodb-kubernetes-operator
      resources:
        limits:
          cpu: 1100m
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 200Mi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-kk4wh
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: k3s-master-3-rock-4aplus
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
      runAsNonRoot: true
      runAsUser: 2000
    serviceAccount: mongodb-kubernetes-operator
    serviceAccountName: mongodb-kubernetes-operator
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: kube-api-access-kk4wh
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2024-03-29T04:06:31Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2024-03-29T04:06:40Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2024-03-29T04:06:40Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2024-03-29T04:06:31Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://c80bcdf9ffc490a9f801730186656917336dd0914c2bd6cb2d754fee202712bf
      image: quay.io/mongodb/mongodb-kubernetes-operator:0.9.0
      imageID: quay.io/mongodb/mongodb-kubernetes-operator@sha256:9f7748cec52db6ef1f413375c625bab0861f63bcfad20f30f332ffc0ebb86795
      lastState: {}
      name: mongodb-kubernetes-operator
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2024-03-29T04:06:39Z"
    hostIP: 192.168.50.12
    phase: Running
    podIP: 10.42.4.143
    podIPs:
    - ip: 10.42.4.143
    qosClass: Burstable
    startTime: "2024-03-29T04:06:31Z"
kind: List
metadata:
  resourceVersion: ""

• The Pod logs:

  • kubectl logs mongo-0

    Defaulted container "mongod" out of: mongod, mongodb-agent, mongod-posthook (init), mongodb-agent-readinessprobe (init)
    2024-03-29T14:36:23.512Z        INFO    versionhook/main.go:32  Running version change post-start hook
    2024-03-29T14:36:23.513Z        INFO    versionhook/main.go:39  Waiting for agent health status...
    2024-03-29T14:36:24.514Z        INFO    versionhook/main.go:75  Pod should not be deleted, mongod started

• The agent clusterconfig of the faulty members:

  • kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/lib/automation/config/cluster-config.json

    {
    "version": 1,
    "processes": [
    {
    "name": "example-mongodb-0",
    "disabled": false,
    "hostname": "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local",
    "args2_6": {
    "net": {
      "port": 27017
    },
    "replication": {
      "replSetName": "example-mongodb"
    },
    "storage": {
      "dbPath": "/data",
      "wiredTiger": {
        "engineConfig": {
          "journalCompressor": "zlib"
        }
      }
    }
    },
    "featureCompatibilityVersion": "7.0",
    "processType": "mongod",
    "version": "7.0.7-jammy",
    "authSchemaVersion": 5,
    "LogRotate": {
    "timeThresholdHrs": 0,
    "sizeThresholdMB": 0
    }
    },
    {
    "name": "example-mongodb-1",
    "disabled": false,
    "hostname": "example-mongodb-1.example-mongodb-svc.mongodb.svc.cluster.local",
    "args2_6": {
    "net": {
      "port": 27017
    },
    "replication": {
      "replSetName": "example-mongodb"
    },
    "storage": {
      "dbPath": "/data",
      "wiredTiger": {
        "engineConfig": {
          "journalCompressor": "zlib"
        }
      }
    }
    },
    "featureCompatibilityVersion": "7.0",
    "processType": "mongod",
    "version": "7.0.7-jammy",
    "authSchemaVersion": 5,
    "LogRotate": {
    "timeThresholdHrs": 0,
    "sizeThresholdMB": 0
    }
    },
    {
    "name": "example-mongodb-2",
    "disabled": false,
    "hostname": "example-mongodb-2.example-mongodb-svc.mongodb.svc.cluster.local",
    "args2_6": {
    "net": {
      "port": 27017
    },
    "replication": {
      "replSetName": "example-mongodb"
    },
    "storage": {
      "dbPath": "/data",
      "wiredTiger": {
        "engineConfig": {
          "journalCompressor": "zlib"
        }
      }
    }
    },
    "featureCompatibilityVersion": "7.0",
    "processType": "mongod",
    "version": "7.0.7-jammy",
    "authSchemaVersion": 5,
    "LogRotate": {
    "timeThresholdHrs": 0,
    "sizeThresholdMB": 0
    }
    }
    ],
    "replicaSets": [
    {
    "_id": "example-mongodb",
    "members": [
    {
      "_id": 0,
      "host": "example-mongodb-0",
      "arbiterOnly": false,
      "votes": 1,
      "priority": 1
    },
    {
      "_id": 1,
      "host": "example-mongodb-1",
      "arbiterOnly": false,
      "votes": 1,
      "priority": 1
    },
    {
      "_id": 2,
      "host": "example-mongodb-2",
      "arbiterOnly": false,
      "votes": 1,
      "priority": 1
    }
    ],
    "protocolVersion": "1",
    "numberArbiters": 0
    }
    ],
    "auth": {
    "usersWanted": [
    {
    "mechanisms": [],
    "roles": [
      {
        "role": "clusterAdmin",
        "db": "admin"
      },
      {
        "role": "userAdminAnyDatabase",
        "db": "admin"
      }
    ],
    "user": "my-user",
    "db": "admin",
    "authenticationRestrictions": [],
    "scramSha256Creds": {
      "iterationCount": 15000,
      "salt": "bWOXC18KXMBmit4HXeCQqSzXZlSRHF5gOFZAXQ==",
      "serverKey": "FzQW2zyJS8ik7o28IpVGhuRUwVaILEg+LgGeFLk9L9k=",
      "storedKey": "c6C4/E73yYPZd7KBKtELbK2Xu7TzhAGd0aijN5LpDAE="
    },
    "scramSha1Creds": {
      "iterationCount": 10000,
      "salt": "ocOdp1rCvtHpDgngqvIWpQ==",
      "serverKey": "iSFG+vjMCoMO0hyCVZDTbGGcFlw=",
      "storedKey": "pAgCAkdWH8yf9kqEKApdPRiFCN8="
    }
    }
    ],
    "disabled": false,
    "authoritativeSet": false,
    "autoAuthMechanisms": [
    "SCRAM-SHA-256"
    ],
    "autoAuthMechanism": "SCRAM-SHA-256",
    "deploymentAuthMechanisms": [
    "SCRAM-SHA-256"
    ],
    "autoUser": "mms-automation",
    "key": "bXwCGvt8m+lrtPdeXjBhLTU+x3japvb/p/hCKOuPqTVj6uwQonCfToiShcoJtuXhFMn0ts2b0w1Qhk5+KX4uFMLh4ZAVeZ6GJP2qN7+QIPZkJsHZIbx57dsBY5ZoJ/JWpx1dWqD6qxRvX+s2uBQe/2BsVTC9Ob3sPW3rlMdCtNn4ldlpA+BcslV448woW3PwQkSIrboqblMgQSgKoOaI2dbMbkgStTBRgQ2ggrlEgNDYk7dcJkVg5DT/6UFdwOWNAlhDv61rTu5Yxt9UJG1CTYPyTvviwbOLIHAElKbCXdakwUysarFdFdnb+rJSxr3D/MsBllvhsiV/h0kmnmmMPNNxY79XLzZ7CXan9GRjIZ7F1pvpN8lCDRXjhZsnt38uxs9/BnpNQ5qAnqvmtN2aYxA7I2newfKI9QG7ekcIOfpysBEpkxlSZz3I1/kYYGzU0eUJDLgr8kjbKlQ/ensseoTJ8rWttTdds6nAkHTqu1TvKMX2PTs+1R9JKWSs3V9KZPDn/+zFTIedBvooyOf/LXU7dgcdvWa0HwEcrIquwQOXDjA48NZyEO5sPAWPRPEQ+0c2gUbFqVcpRMlnc8KS/UwXAOts3Rh1VP6x43q73FCZfHLVCwJGXjkVZ3JfxL+C0lHddqCeSEdFCZ/dqTpzUM30mv4=",
    "keyfile": "/var/lib/mongodb-mms-automation/authentication/keyfile",
    "keyfileWindows": "%SystemDrive%\\MMSAutomation\\versions\\keyfile",
    "autoPwd": "6qL8olnmrw_grZao6bXP"
    },
    "tls": {
    "CAFilePath": "",
    "clientCertificateMode": "OPTIONAL"
    },
    "mongoDbVersions": [
    {
    "name": "7.0.7-jammy",
    "builds": [
    {
      "platform": "linux",
      "url": "",
      "gitVersion": "",
      "architecture": "amd64",
      "flavor": "rhel",
      "minOsVersion": "",
      "maxOsVersion": "",
      "modules": []
    },
    {
      "platform": "linux",
      "url": "",
      "gitVersion": "",
      "architecture": "amd64",
      "flavor": "ubuntu",
      "minOsVersion": "",
      "maxOsVersion": "",
      "modules": []
    },
    {
      "platform": "linux",
      "url": "",
      "gitVersion": "",
      "architecture": "aarch64",
      "flavor": "ubuntu",
      "minOsVersion": "",
      "maxOsVersion": "",
      "modules": []
    },
    {
      "platform": "linux",
      "url": "",
      "gitVersion": "",
      "architecture": "aarch64",
      "flavor": "rhel",
      "minOsVersion": "",
      "maxOsVersion": "",
      "modules": []
    }
    ]
    }
    ],
    "backupVersions": [],
    "monitoringVersions": [],
    "options": {
    "downloadBase": "/var/lib/mongodb-mms-automation"
    }
    }

• The agent health status of the faulty members:

  • kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json

    {
    "statuses": {
    "example-mongodb-0": {
    "IsInGoalState": false,
    "LastMongoUpTime": 0,
    "ExpectedToBeUp": true,
    "ReplicationStatus": -1
    }
    },
    "mmsStatus": {
    "example-mongodb-0": {
    "name": "example-mongodb-0",
    "lastGoalVersionAchieved": -1,
    "plans": [
    {
      "automationConfigVersion": 1,
      "started": "2024-03-29T14:34:42.321636573Z",
      "completed": null,
      "moves": [
        {
          "move": "Start",
          "moveDoc": "Start the process",
          "steps": [
            {
              "step": "StartFresh",
              "stepDoc": "Start a mongo instance  (start fresh)",
              "isWaitStep": false,
              "started": "2024-03-29T14:34:42.321737784Z",
              "completed": null,
              "result": ""
            }
          ]
        },
        {
          "move": "WaitAllRsMembersUp",
          "moveDoc": "Wait until all members of this process' repl set are up",
          "steps": [
            {
              "step": "WaitAllRsMembersUp",
              "stepDoc": "Wait until all members of this process' repl set are up",
              "isWaitStep": true,
              "started": null,
              "completed": null,
              "result": ""
            }
          ]
        },
        {
          "move": "RsInit",
          "moveDoc": "Initialize a replica set including the current MongoDB process",
          "steps": [
            {
              "step": "RsInit",
              "stepDoc": "Initialize a replica set",
              "isWaitStep": false,
              "started": null,
              "completed": null,
              "result": ""
            }
          ]
        },
        {
          "move": "WaitFeatureCompatibilityVersionCorrect",
          "moveDoc": "Wait for featureCompatibilityVersion to be right",
          "steps": [
            {
              "step": "WaitFeatureCompatibilityVersionCorrect",
              "stepDoc": "Wait for featureCompatibilityVersion to be right",
              "isWaitStep": true,
              "started": null,
              "completed": null,
              "result": ""
            }
          ]
        }
      ]
    }
    ],
    "errorCode": 0,
    "errorString": ""
    }
    }
    }

[kennedyoliveira]

• The verbose agent logs of the faulty members:
  • kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent-verbose.log

[2024-03-29T04:09:11.779+0000] [header.info] [::0]        GitCommitId = 956e3386ad456471db1776d79637a38f182a6088
[2024-03-29T04:09:11.779+0000] [header.info] [::0]  AutomationVersion = 107.0.0.8465
[2024-03-29T04:09:11.780+0000] [header.info] [::0]          localhost = example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local
[2024-03-29T04:09:11.780+0000] [header.info] [::0] ErrorStateSleepTime = 10s
[2024-03-29T04:09:11.780+0000] [header.info] [::0] GoalStateSleepTime = 10s
[2024-03-29T04:09:11.780+0000] [header.info] [::0] NotGoalStateSleepTime = 1s
[2024-03-29T04:09:11.780+0000] [header.info] [::0]     PlanCutoffTime = 300000
[2024-03-29T04:09:11.780+0000] [header.info] [::0]       TracePlanner = false
[2024-03-29T04:09:11.780+0000] [header.info] [::0]               User = mongodb
[2024-03-29T04:09:11.780+0000] [header.info] [::0]         Go version = go1.20.10
[2024-03-29T04:09:11.780+0000] [header.info] [::0]         MmsBaseURL = 
[2024-03-29T04:09:11.780+0000] [header.info] [::0]         MmsGroupId = 
[2024-03-29T04:09:11.780+0000] [header.info] [::0]          HttpProxy = 
[2024-03-29T04:09:11.780+0000] [header.info] [::0] DisableHttpKeepAlive = false
[2024-03-29T04:09:11.780+0000] [header.info] [::0]        HttpsCAFile = 
[2024-03-29T04:09:11.780+0000] [header.info] [::0] TlsRequireValidMMSServerCertificates = true
[2024-03-29T04:09:11.780+0000] [header.info] [::0] TlsMMSServerClientCertificate = 
[2024-03-29T04:09:11.780+0000] [header.info] [::0] KMIPProxyCertificateDir = /tmp
[2024-03-29T04:09:11.780+0000] [header.info] [::0] EnableLocalConfigurationServer = false
[2024-03-29T04:09:11.780+0000] [header.info] [::0] DialTimeoutSeconds = 40
[2024-03-29T04:09:11.780+0000] [header.info] [::0] KeepUnusedMongodbVersions = false
[2024-03-29T04:09:11.780+0000] [header.info] [::0] DisallowDowngrades = false
[2024-03-29T04:09:11.781+0000] [.info] [src/util/sysdep_unix.go:LockAutomationLockFile:322] [04:09:11.781] Locking automation lock file at /tmp/mongodb-mms-automation.lock
[2024-03-29T04:09:11.781+0000] [.info] [main/components/agent.go:NewAgent:132] [04:09:11.781] Constructing new agent object with desiredClusterConfigPath=/var/lib/automation/config/cluster-config.json
[2024-03-29T04:09:11.782+0000] [.info] [src/mongosqld/custodian.go:NewCustodian:141] <mongosqld custodian> [04:09:11.782] Started
[2024-03-29T04:09:11.782+0000] [.warn] [prometheus/collector/hardwarecollector.go:collectMetrics:77] <promHardwareCollector> [04:09:11.782] agent.MetricsCollector is not running. Check agent.go for initialization issues : collector is not running
[2024-03-29T04:09:11.782+0000] [.info] [src/config/config.go:ReadClusterConfig:390] [04:09:11.782] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2024-03-29T04:09:11.787+0000] [.info] [realtime/rtcollector/rtcollector.go:controlLoop:85] <rtCollector> [04:09:11.786] Starting control loop
[2024-03-29T04:09:11.788+0000] [.info] [src/dataexplorer/dataexplorer.go:controlLoop:109] <dataExplorer> [04:09:11.787] Starting control loop
[2024-03-29T04:09:11.788+0000] [.info] [src/kmipproxy/custodian.go:mainLoop:204] <kmipProxyMaintainer> [04:09:11.787] Starting main loop
[2024-03-29T04:09:11.891+0000] [.info] [main/components/agent.go:LoadClusterConfig:273] [04:09:11.891] New cluster config received! 0 (<nil>) -> 1 (2024-03-29 04:08:27.397110092 +0000 UTC)
[2024-03-29T04:09:11.891+0000] [.info] [src/modules/agents_unix.go:KillAllAgents:43] [04:09:11.891] Killing all running mongodb-mms-monitoring-agent agents at /var/lib/mongodb-mms-automation/mongodb-mms-monitoring-agent-.+\..+_.+/mongodb-mms-monitoring-agent *$
[2024-03-29T04:09:11.977+0000] [.info] [src/modules/agents_unix.go:KillAllAgents:43] [04:09:11.977] Killing all running mongodb-mms-backup-agent agents at /var/lib/mongodb-mms-automation/mongodb-mms-backup-agent-.+\..+_.+/mongodb-mms-backup-agent *$
[2024-03-29T04:09:12.325+0000] [.info] [main/components/agent.go:launchNewDirectors:568] [04:09:12.325] Launched director example-mongodb-0
[2024-03-29T04:09:12.326+0000] [.info] [metrics/collector/realcollector.go:collectLoop:308] <hardwareMetricsCollector> [04:09:12.325] Received new cluster config version 1, edition 2024-03-29 04:08:27.397110092 +0000 UTC!
[2024-03-29T04:09:12.326+0000] [.info] [src/dataexplorer/dataexplorer.go:controlLoop:115] <dataExplorer> [04:09:12.326] Received new cluster config version 1, edition 2024-03-29 04:08:27.397110092 +0000 UTC!
[2024-03-29T04:09:12.326+0000] [.info] [src/director/director.go:incorporateConfig:302] <example-mongodb-0> [04:09:12.326] clusterConfig edition is different (<nil> -> 2024-03-29 04:08:27.397110092 +0000 UTC).  Incorporating...
[2024-03-29T04:09:12.326+0000] [.info] [metrics/prometheus/exporter.go:mainLoop:85] <prometheusExporter> [04:09:12.326] Received new cluster config version 1, edition 2024-03-29 04:08:27.397110092 +0000 UTC!
[2024-03-29T04:09:12.326+0000] [.info] [metrics/prometheus/server.go:ApplyNewPromConfig:70] <prometheusExporter> [04:09:12.326] Prometheus config is nil or has prometheus explicitly disabled, stopping the prometheus server if running...
[2024-03-29T04:09:12.326+0000] [.info] [metrics/prometheus/server.go:func1:68] <prometheusExporter> [04:09:12.326] ApplyNewPromConfig took 68.543┬╡s
[2024-03-29T04:09:12.326+0000] [.info] [realtime/rtcollector/rtcollector.go:controlLoop:93] <rtCollector> [04:09:12.326] Received new cluster config version 1, edition 2024-03-29 04:08:27.397110092 +0000 UTC!
[2024-03-29T04:09:12.354+0000] [.error] [main/components/agent.go:ApplyClusterConfig:358] [04:09:12.354] Log path absent for process=state.ProcessConfigName=example-mongodb-0ProcessType=mongodVersion=7.0.7-jammyFullVersion={"trueName":"7.0.7-jammy","gitVersion":"","modules":[],"major":7,"minor":0,"patch":7}Disabled=falseManualMode=falseNumCores=0CpuAffinity=[]LogRotate={"sizeThresholdMB":0,"timeThresholdHrs":0,"numUncompressed":0,"numTotal":0,"percentOfDiskspace":0,"includeAuditLogsWithMongoDBLogs":false}AuditLogRotate=<nil>LastResync="0001-01-01T00:00:00Z"LastThirdPartyRestoreResync="0001-01-01T00:00:00Z"LastCompact="0001-01-01T00:00:00Z"LastKmipMasterKeyRotation="0001-01-01T00:00:00Z"LastRestart="0001-01-01T00:00:00Z"Hostname=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.localAlias=Cluster=AuthSchemaVersion=5FeatureCompatibilityVersion=7.0Kerberos=<nil>Args={"net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"example-mongodb"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}ProcessAuthInfo={"UsersWanted":[{"user":"my-user","db":"admin","authenticationRestrictions":[],"scramSha1Creds":{"iterationCount":10000,"salt":"kdyZWmkUcb3a8ZmcNW/pAQ==","storedKey":"f+F8brZw6+svCr5npW9pQdt9Y+0=","serverKey":"BYni3jThRK7cui+qSaA2lXS7lxo="},"scramSha256Creds":{"iterationCount":15000,"salt":"5y33Cj4wFtoGnXlUziSDus8SkAqkLiHXk4QVVQ==","storedKey":"v+6RTSIsZQprrgUntjz+1Ix3aj1NXATr4MGnqGzDc9Q=","serverKey":"IRFx8w6rmq1Ak021Afsw/hB6Q8P2I6MC4jzqGKKnVEs="},"roles":[{"role":"clusterAdmin","db":"admin","minFcv":""},{"role":"userAdminAnyDatabase","db":"admin","minFcv":""}],"inheritedRoles":null,"mechanisms":[],"scope":null}],"UsersDeleted":null,"Roles":null,"DesiredKey":"[ZpnXOsjiRNY-REDACTED-G04AKp5vLX0]","DesiredNewKey":"[ZpnXOsjiRNY-REDACTED-G04AKp5vLX0]","DesiredKeyHash":"A0/2wZ9u8QjNb3N4O0PoarnF2dQthv+qEnkY+Q++W4A=","DesiredNewKeyHash":null,"KeyfileHashes":["A0/2wZ9u8QjNb3N4O0PoarnF2dQthv+qEnkY+Q++W4A="],"UsingAuth":true}IsConfigServer=falseIsShardServer=falseIsInReplSet=trueIsStandalone=falseIsArbiter=falseDownloadBase=/var/lib/mongodb-mms-automationFullySyncRsTags=falseReplicaSetId=example-mongodbBackupRestoreUrl=<redacted>, BackupRestoreUrlV3=BackupParallelRestoreUrl=BackupParallelRestoreNumChunks=0BackupParallelRestoreNumWorkers=0BackupThirdPartyRestoreBaseUrl=BackupRestoreRsVersion=0BackupRestoreElectionTerm=0BackupRestoreCheckpointTimestamp=<nil>BackupRestoreCertificateValidationHostname=BackupRestoreSystemUsersUUID=BackupRestoreSystemRolesUUID=BackupRestoreBalancerSettings=nullBackupRestoreConfigSettingsUUID=BackupShardIdRestoreMaps=[]DirectAttachVerificationKey=DirectAttachSourceClusterName=DirectAttachShouldFilterByFileList=falseConfigPath=StorageEngine=BackupRestoreOplogBaseUrl=BackupRestoreOplog=<nil>BackupRestoreDesiredTime=<nil>BackupRestoreSourceRsId=BackupRestoreFilterList=<nil>BackupRestoreFilteredFileListUrl=BackupRestoreJobId=BackupRestoreVerificationKey=BackupRestoreSourceGroupId=PitRestoreType=BackupThirdPartyOplogStoreType=EncryptionProviderType=KMIPProxyPort=0KMIPProxyDisabled=falseTemporaryPort=0CredentialsVersion=0Repair=nullRealtimeConfig=<nil>DataExplorerConfig=<nil>DefaultRWConcern=<nil>LdapCaPath=ConfigServers=[]RestartIntervalTimeMs=<nil>ClusterWideConfiguration=ProfilingConfig=<nil>RegionBaseUrl=RegionBaseRealtimeUrl=RegionBaseAgentUrl=StepDownPrimaryForResync=falsekey=<nil>keyLock=null. log destination=
[2024-03-29T04:09:12.355+0000] [.error] [src/main/cm.go:mainLoop:520] [04:09:12.355] Error applying desired cluster configs : [04:09:12.354] Log path absent for process=state.ProcessConfigName=example-mongodb-0ProcessType=mongodVersion=7.0.7-jammyFullVersion={"trueName":"7.0.7-jammy","gitVersion":"","modules":[],"major":7,"minor":0,"patch":7}Disabled=falseManualMode=falseNumCores=0CpuAffinity=[]LogRotate={"sizeThresholdMB":0,"timeThresholdHrs":0,"numUncompressed":0,"numTotal":0,"percentOfDiskspace":0,"includeAuditLogsWithMongoDBLogs":false}AuditLogRotate=<nil>LastResync="0001-01-01T00:00:00Z"LastThirdPartyRestoreResync="0001-01-01T00:00:00Z"LastCompact="0001-01-01T00:00:00Z"LastKmipMasterKeyRotation="0001-01-01T00:00:00Z"LastRestart="0001-01-01T00:00:00Z"Hostname=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.localAlias=Cluster=AuthSchemaVersion=5FeatureCompatibilityVersion=7.0Kerberos=<nil>Args={"net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"example-mongodb"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}ProcessAuthInfo={"UsersWanted":[{"user":"my-user","db":"admin","authenticationRestrictions":[],"scramSha1Creds":{"iterationCount":10000,"salt":"kdyZWmkUcb3a8ZmcNW/pAQ==","storedKey":"f+F8brZw6+svCr5npW9pQdt9Y+0=","serverKey":"BYni3jThRK7cui+qSaA2lXS7lxo="},"scramSha256Creds":{"iterationCount":15000,"salt":"5y33Cj4wFtoGnXlUziSDus8SkAqkLiHXk4QVVQ==","storedKey":"v+6RTSIsZQprrgUntjz+1Ix3aj1NXATr4MGnqGzDc9Q=","serverKey":"IRFx8w6rmq1Ak021Afsw/hB6Q8P2I6MC4jzqGKKnVEs="},"roles":[{"role":"clusterAdmin","db":"admin","minFcv":""},{"role":"userAdminAnyDatabase","db":"admin","minFcv":""}],"inheritedRoles":null,"mechanisms":[],"scope":null}],"UsersDeleted":null,"Roles":null,"DesiredKey":"[ZpnXOsjiRNY-REDACTED-G04AKp5vLX0]","DesiredNewKey":"[ZpnXOsjiRNY-REDACTED-G04AKp5vLX0]","DesiredKeyHash":"A0/2wZ9u8QjNb3N4O0PoarnF2dQthv+qEnkY+Q++W4A=","DesiredNewKeyHash":null,"KeyfileHashes":["A0/2wZ9u8QjNb3N4O0PoarnF2dQthv+qEnkY+Q++W4A="],"UsingAuth":true}IsConfigServer=falseIsShardServer=falseIsInReplSet=trueIsStandalone=falseIsArbiter=falseDownloadBase=/var/lib/mongodb-mms-automationFullySyncRsTags=falseReplicaSetId=example-mongodbBackupRestoreUrl=<redacted>, BackupRestoreUrlV3=BackupParallelRestoreUrl=BackupParallelRestoreNumChunks=0BackupParallelRestoreNumWorkers=0BackupThirdPartyRestoreBaseUrl=BackupRestoreRsVersion=0BackupRestoreElectionTerm=0BackupRestoreCheckpointTimestamp=<nil>BackupRestoreCertificateValidationHostname=BackupRestoreSystemUsersUUID=BackupRestoreSystemRolesUUID=BackupRestoreBalancerSettings=nullBackupRestoreConfigSettingsUUID=BackupShardIdRestoreMaps=[]DirectAttachVerificationKey=DirectAttachSourceClusterName=DirectAttachShouldFilterByFileList=falseConfigPath=StorageEngine=BackupRestoreOplogBaseUrl=BackupRestoreOplog=<nil>BackupRestoreDesiredTime=<nil>BackupRestoreSourceRsId=BackupRestoreFilterList=<nil>BackupRestoreFilteredFileListUrl=BackupRestoreJobId=BackupRestoreVerificationKey=BackupRestoreSourceGroupId=PitRestoreType=BackupThirdPartyOplogStoreType=EncryptionProviderType=KMIPProxyPort=0KMIPProxyDisabled=falseTemporaryPort=0CredentialsVersion=0Repair=nullRealtimeConfig=<nil>DataExplorerConfig=<nil>DefaultRWConcern=<nil>LdapCaPath=ConfigServers=[]RestartIntervalTimeMs=<nil>ClusterWideConfiguration=ProfilingConfig=<nil>RegionBaseUrl=RegionBaseRealtimeUrl=RegionBaseAgentUrl=StepDownPrimaryForResync=falsekey=<nil>keyLock=null. log destination=
[2024-03-29T04:09:12.439+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:12.439] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:12.439] Non-TLS attempt failed : [04:09:12.438] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:12.730+0000] [.info] [state2/currentstate/repl.go:getRsStatusFromOtherMember:558] <example-mongodb-0> [04:09:12.729] rs member = example-mongodb-1.example-mongodb-svc.mongodb.svc.cluster.local:27017 is down. Skipping...
[2024-03-29T04:09:12.790+0000] [.info] [src/runtimestats/printer.go:mainLoop:190] <runtimestatsPrinter> [04:09:12.790] stats:rss=47MB   heapInUse=9MB   memoryUsage=1.24%   normalizedCPU=0.00% absoluteCPU=0.00%   gcNum=3 totalGcPauseNs=678728
[2024-03-29T04:09:12.824+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:12.824] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:12.825+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:12.825] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:12.928+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:12.928] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:12.928] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:13.037+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:13.037] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:13.037] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:13.075+0000] [.info] [state2/currentstate/repl.go:getRsStatusFromOtherMember:558] <example-mongodb-0> [04:09:13.075] rs member = example-mongodb-2.example-mongodb-svc.mongodb.svc.cluster.local:27017 is down. Skipping...
[2024-03-29T04:09:13.140+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:13.140] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:13.140] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:13.141+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:13.141] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:13.141] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:13.200+0000] [.info] [src/director/director.go:computePlan:279] <example-mongodb-0> [04:09:13.200] ... process has a plan : Start,WaitAllRsMembersUp,RsInit,WaitFeatureCompatibilityVersionCorrect
[2024-03-29T04:09:13.200+0000] [.info] [src/director/director.go:tracef:813] <example-mongodb-0> [04:09:13.200] Running step: 'StartFresh' of move 'Start'
[2024-03-29T04:09:13.200+0000] [.info] [src/director/director.go:tracef:813] <example-mongodb-0> [04:09:13.200] because 
[All the following are true: 
    ['currentState.Running' = false]
    ['currentState.IsVCRedistCorrect' = true]
    ['desiredState.ProcessType' != mongos ('desiredState.ProcessType' = mongod)]
    ['currentState.StorageFieldsChangeFinished' = true]
    ['desiredState.GetRestoreKey()' = ]
    ['desiredState.IsThirdPartyRestoreResync' = false]
    ['currentState.CanStartFresh' = true]
]
[2024-03-29T04:09:13.204+0000] [.info] [src/config/config.go:createKeyFile:2909] <example-mongodb-0> [04:09:13.203] Creating key file '/var/lib/mongodb-mms-automation/authentication/keyfile' keyfileMode=...
[2024-03-29T04:09:13.204+0000] [.info] [src/util/file.go:CreateFileAndParentDirectories:498] <example-mongodb-0> [04:09:13.204] Creating file /var/lib/mongodb-mms-automation/authentication/keyfile with permissions -rw-------
[2024-03-29T04:09:13.306+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:13.306] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:13.306] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:13.309+0000] [.info] [src/util/sysdep_unix.go:MachineHasNumactl:635] <example-mongodb-0> [04:09:13.309] Got err = exec: "numactl": executable file not found in $PATH running numactl cmd = Args=[numactl --show] (output=). Assuming machine does not have numactl installed.
[2024-03-29T04:09:13.309+0000] [.info] [src/mongoctl/processctl.go:newStartProcessCmd:589] <example-mongodb-0> [04:09:13.309] Running command mongod with args [-f /data/automation-mongod.conf]
[2024-03-29T04:09:13.412+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:13.412] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:13.412] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:13.547+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:13.547] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:13.547] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:13.681+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:13.681] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:13.681] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:13.803+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:13.803] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:13.804+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:13.804] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:13.815+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:13.815] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:13.815] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:13.921+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:13.921] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:13.921] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:13.949+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:13.949] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:13.949] Non-TLS attempt failed : [04:09:13.949] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:14.025+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:14.025] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:14.025] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:14.027+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:14.027] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:14.027] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:14.084+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:14.084] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:14.083] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:14.218+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:14.218] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:14.218] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:14.352+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:14.352] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:14.352] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:14.486+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:14.486] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:14.486] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:14.620+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:14.620] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:14.619] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:14.754+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:14.754] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:14.754] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:14.818+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:14.817] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:14.819+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:14.818] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:14.888+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:14.887] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:14.887] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:14.991+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:14.991] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:14.991] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.021+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.021] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.021] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.096+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.096] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.096] Non-TLS attempt failed : [04:09:15.096] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.097+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:15.097] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:15.097] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:15.155+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.155] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.154] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.290+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.290] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.290] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.423+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.423] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.423] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.557+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.557] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.556] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.690+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.690] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.690] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.815+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:15.815] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:15.815+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:15.815] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:15.825+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.825] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.825] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.934+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.934] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.934] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:15.962+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:15.962] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:15.962] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:16.041+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:16.041] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:16.041] Non-TLS attempt failed : [04:09:16.041] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:16.042+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:16.042] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:16.042] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:16.097+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:16.097] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:16.097] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:16.231+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:16.231] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:16.231] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:16.364+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:16.364] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:16.364] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:16.499+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:16.499] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:16.499] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:16.633+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:16.633] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:16.633] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:16.767+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:16.767] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:16.766] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:16.820+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:16.820] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:16.821+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:16.821] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:16.901+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:16.901] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:16.901] Non-TLS attempt failed : [04:09:16.901] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.009+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.009] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.009] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.039+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.039] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.038] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.115+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.115] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.115] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.116+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:17.116] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:17.116] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:17.181+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.181] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.181] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.315+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.315] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.315] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.449+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.448] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.448] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.583+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.583] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.583] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.717+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.716] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.716] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.811+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:17.811] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:17.811+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:17.811] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:17.851+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.851] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.851] Non-TLS attempt failed : [04:09:17.851] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.958+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.958] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.958] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:17.987+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:17.987] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:17.987] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:18.061+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:18.061] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:18.061] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:18.062+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:18.062] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:18.062] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:18.121+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:18.121] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:18.120] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:18.253+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:18.253] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:18.253] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:18.387+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:18.387] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:18.387] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:18.388+0000] [.info] [src/process/process.go:SleepUntilTrue:44] <example-mongodb-0> [04:09:18.388] Still sleeping until start.go's sleepUntilProcessUp (function com.tengen/cm/action.sleepUntilProcessUp.func1) returns true.  It's been 4.9s.
[2024-03-29T04:09:18.522+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:18.522] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:18.522] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:18.656+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:18.656] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:18.655] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:18.790+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:18.789] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:18.789] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:18.816+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:18.816] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:18.818+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:18.818] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:18.925+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:18.925] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:18.925] Non-TLS attempt failed : [04:09:18.925] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.028+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.028] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.028] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.059+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.059] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.058] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.132+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.132] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.132] Non-TLS attempt failed : [04:09:19.132] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.133+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:19.133] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:19.133] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:19.193+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.193] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.193] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.326+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.326] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.326] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.460+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.459] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.459] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.594+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.594] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.593] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.731+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.731] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.730] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.826+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:19.826] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:19.826+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:19.826] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:19.867+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.867] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.866] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:19.974+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:19.974] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:19.974] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:20.004+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:20.004] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:20.003] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:20.079+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:20.079] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:20.079] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:20.083+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:20.083] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:20.083] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:20.139+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:20.139] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:20.139] Non-TLS attempt failed : [04:09:20.139] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:20.272+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:20.272] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:20.271] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:20.405+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:20.405] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:20.405] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:20.538+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:20.538] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:20.538] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:20.672+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:20.672] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:20.671] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:20.806+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:20.806] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:20.805] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:20.818+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:20.818] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:20.818+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:20.818] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:20.933+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:20.933] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:20.933] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.039+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.039] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.039] Non-TLS attempt failed : [04:09:21.039] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.070+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.070] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.070] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.142+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.142] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.141] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.143+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:21.143] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:21.143] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:21.203+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.203] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.203] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.337+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.337] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.337] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.470+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.470] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.470] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.604+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.604] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.604] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.738+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.738] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.738] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.823+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:21.823] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:21.823+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:21.823] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:21.874+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.874] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.873] Non-TLS attempt failed : [04:09:21.873] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:21.977+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:21.977] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:21.977] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.010+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.010] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.010] Non-TLS attempt failed : [04:09:22.010] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.082+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.082] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.082] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.083+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:22.083] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:22.083] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:22.146+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.146] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.146] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.279+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.279] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.279] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.375+0000] [.info] [src/config/config.go:ReadClusterConfig:390] [04:09:22.375] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2024-03-29T04:09:22.375+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [04:09:22.375] clusterConfig unchanged
[2024-03-29T04:09:22.413+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.413] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.413] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.547+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.547] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.547] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.681+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.681] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.681] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.814+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:22.814] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:22.814+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:22.814] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:22.816+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.816] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.815] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.920+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.920] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.920] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:22.953+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:22.953] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:22.953] Non-TLS attempt failed : [04:09:22.953] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:23.024+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:23.024] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:23.024] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:23.026+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:23.026] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:23.026] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:23.087+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:23.087] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:23.087] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:23.221+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:23.221] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:23.221] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:23.356+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:23.355] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:23.355] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:23.384+0000] [.info] [src/config/config.go:ReadClusterConfig:390] [04:09:23.384] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2024-03-29T04:09:23.385+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [04:09:23.385] clusterConfig unchanged
[2024-03-29T04:09:23.489+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:23.489] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:23.488] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:23.489+0000] [.info] [src/process/process.go:SleepUntilTrue:44] <example-mongodb-0> [04:09:23.489] Still sleeping until start.go's sleepUntilProcessUp (function com.tengen/cm/action.sleepUntilProcessUp.func1) returns true.  It's been 10s.
[2024-03-29T04:09:23.623+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:23.623] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:23.623] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:23.757+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:23.757] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:23.757] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:23.815+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:23.815] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:23.815+0000] [.info] [metrics/disk/disk_unix.go:runLsblkCommand:436] <hardwareMetricsCollector> [04:09:23.815] lsblk cache invalidated, running lsblk command
[2024-03-29T04:09:23.894+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:23.894] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:23.894] Non-TLS attempt failed : [04:09:23.894] context canceled : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:23.997+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:23.997] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:23.997] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:24.028+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:24.028] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:24.028] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:24.100+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:24.100] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:24.100] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:24.101+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [04:09:24.101] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [04:09:24.101] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T04:09:24.163+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:24.163] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:24.163] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:24.296+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [04:09:24.296] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[04:09:24.296] Non-TLS attempt failed : dial tcp 10.42.5.23:27017: connect: connection refused
[2024-03-29T04:09:24.424+0000] [.info] [src/config/config.go:ReadClusterConfig:390] [04:09:24.424] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2024-03-29T14:36:43.491+0000] [.info] [src/config/config.go:ReadClusterConfig:390] [14:36:43.490] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2024-03-29T14:36:43.491+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [14:36:43.491] clusterConfig unchanged
[2024-03-29T14:36:43.531+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [14:36:43.531] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[14:36:43.531] Non-TLS attempt failed : dial tcp 10.42.5.30:27017: connect: connection refused
[2024-03-29T14:36:43.532+0000] [.warn] [metrics/collector/util.go:getPingStatus:127] <hardwareMetricsCollector> [14:36:43.532] Failed to fetch replStatus for example-mongodb-0 : <hardwareMetricsCollector> [14:36:43.532] Server at example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) is down
[2024-03-29T14:36:43.589+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [14:36:43.589] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[14:36:43.589] Non-TLS attempt failed : dial tcp 10.42.5.30:27017: connect: connection refused
[2024-03-29T14:36:43.723+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [14:36:43.723] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[14:36:43.723] Non-TLS attempt failed : dial tcp 10.42.5.30:27017: connect: connection refused
[2024-03-29T14:36:43.860+0000] [.warn] [src/mongoclientservice/mongoclientservice.go:createClient:1238] [14:36:43.860] Server for client=0x0 connParams=example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017 (local=false) determined to be down due to error=[14:36:43.860] Non-TLS attempt failed : [14:36:43.860] context canceled : dial tcp 10.42.5.30:27017: connect: connection refused

[github-actions[bot]]

This issue is being marked stale because it has been open for 60 days with no activity. Please comment if this issue is still affecting you. If there is no change, this issue will be closed in 30 days.