mongodb / mongodb-kubernetes-operator

MongoDB Community Kubernetes Operator
Other
1.18k stars 480 forks source link

Readiness probe failed: command "/opt/scripts/readinessprobe" timed out on DB Pod when installing on Mac M1, even when using arm64 images #1420

Open vinnytwice opened 8 months ago

vinnytwice commented 8 months ago

What did you do to encounter the bug? Steps to reproduce the behavior:

  1. Install community operator via helm chart
  2. Install mongo community edition via (custom) helm chart
  3. run command kubectl get po -w
  4. NAME                                           READY   STATUS    RESTARTS   AGE
    mongodb-kubernetes-operator-6db4f65679-4prl9   1/1     Running   0          73m
    mongo-rs-0                                     1/2     Running   0          29m
  5. run command describe po mongo-rs-0

What did you expect? MongoDB rs pod running as it does when I install the same charts on Hetzner Cloud (dough on x86 processor) running rancher/k3s:v1.27.4-k3s1 image. The only difference is that on local K3D cluster I declare PVs for physical external ssd, and the SC provisions them instead of provisioning the cloud provider's.

What happened instead? mongo-rs-0 is stuck in 1/2 ready state, dough PVs are created and data and logs are populated in the disk.

Things I tried: Overriding the mongo-community-operator helm chart, pointing directly to arm64 images for the operator an the redness probe ( couldn't find an arm64 image for the agent) , using a custom values.yaml:

 operator:
  name: mongodb-kubernetes-operator
  # Name of the operator image
  operatorImageName: mongodb-kubernetes-operator
  # Name of the deployment of the operator pod
  deploymentName: mongodb-kubernetes-operator
  # Version of mongodb-kubernetes-operator
  # version: 0.8.3
  version: 0.8.3-arm64
  # watchNamespace: "*"
  resources:
    limits:
      cpu: 200m #1100m
      memory: 200Mi #1Gi
    requests:
      cpu: 200m #500m
      memory: 200Mi #500Mi
agent:
  name: mongodb-agent
  version: 12.0.25.7724-1
versionUpgradeHook:
  name: mongodb-kubernetes-operator-version-upgrade-post-start-hook
  version: 1.0.8
readinessProbe:
  name: mongodb-kubernetes-readinessprobe
  # version: 1.0.17 #(mongo error: Readiness probe failed: command "/opt/scripts/readinessprobe" timed out  )
  version: 1.0.17-arm64 #(mongo error: Readiness probe failed: panic: open /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json: no such file or directory )
mongodb:
  name: mongo
  repo: docker.io

But the reason of the readiness prob failing is now Readiness probe failed: panic: open /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json: no such file or directory

Screenshots If applicable, add screenshots to help explain your problem. Schermata 2023-11-01 alle 15 40 58 Schermata 2023-11-01 alle 15 40 47

Operator Information I tried different combinations of operator and db version

Kubernetes Cluster Information K3D local cluster running rancher/k3s:v1.27.4-k3s1 image, on MacBook Air M1 with external ssd used for PVs

**kubectl describe output***

Name:             mongo-rs-0
Namespace:        default
Priority:         0
Service Account:  mongodb-database
Node:             docker-desktop/192.168.65.3
Start Time:       Thu, 02 Nov 2023 10:24:40 +0100
Labels:           app=mongo-rs-svc
                  controller-revision-hash=mongo-rs-7c746d48d7
                  statefulset.kubernetes.io/pod-name=mongo-rs-0
Annotations:      agent.mongodb.com/version: 1
Status:           Running
IP:               10.1.1.177
IPs:
  IP:           10.1.1.177
Controlled By:  StatefulSet/mongo-rs
Init Containers:
  mongod-posthook:
    Container ID:  docker://41885b3e6f3d6318855be29896e7704bacb0e8a49d2cde2e70989b370b730cfc
    Image:         quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
    Image ID:      docker-pullable://quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:5ebea828235618936f059a89048098fca662cd665a0589ea25f5cbee2676a2ed
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      version-upgrade-hook
      /hooks/version-upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 02 Nov 2023 10:24:42 +0100
      Finished:     Thu, 02 Nov 2023 10:24:42 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /hooks from hooks (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tpff9 (ro)
  mongodb-agent-readinessprobe:
    Container ID:  docker://b209116d6165333f5a3a1e743938e181385e8c4e64d93c41d677dfb9117b95d5
    Image:         quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17-arm64
    Image ID:      docker-pullable://quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:43b2bba2b7d7ab157d707f365bfaf7a8eb080969a025af09be99c3885e583556
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      /probes/readinessprobe
      /opt/scripts/readinessprobe
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 02 Nov 2023 10:24:44 +0100
      Finished:     Thu, 02 Nov 2023 10:24:44 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /opt/scripts from agent-scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tpff9 (ro)
Containers:
  mongod:
    Container ID:  docker://e15ca7325fb920973eb516b50c35819762bb648a43aff4b58c4b76ec5ac94e4d
    Image:         docker.io/mongo:6.0.11
    Image ID:      docker-pullable://mongo@sha256:9f107d34c284d9be2527341d6d511cd818a569d83b4c52d1aef4ca7a75c487d7
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c

      #run post-start hook to handle version changes
      /hooks/version-upgrade

      # wait for config and keyfile to be created by the agent
       while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

      # start mongod with this configuration
      exec mongod -f /data/automation-mongod.conf;

    Args:

    State:          Running
      Started:      Thu, 02 Nov 2023 10:25:01 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  214748364800m
    Requests:
      cpu:     200m
      memory:  214748364800m
    Environment:
      AGENT_STATUS_FILEPATH:  /healthstatus/agent-health-status.json
    Mounts:
      /data from data-volume (rw)
      /healthstatus from healthstatus (rw)
      /hooks from hooks (rw)
      /tmp from tmp (rw)
      /var/lib/mongodb-mms-automation/authentication from mongo-rs-keyfile (rw)
      /var/log/mongodb-mms-automation from logs-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tpff9 (ro)
  mongodb-agent:
    Container ID:  docker://cb14d9703fd860a63636c54b7bf4bc2afc32fa1d94dc79aa04bb852a6acf5e05
    Image:         quay.io/mongodb/mongodb-agent:12.0.25.7724-1
    Image ID:      docker-pullable://quay.io/mongodb/mongodb-agent@sha256:ae9a33c87fc623985b26140ed93bb2ea510fd3682279ea50770f9c4d47b20b2c
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
      current_uid=$(id -u)
      AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
      declare -r current_uid
      if ! grep -q "${current_uid}" /etc/passwd ; then
      sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
      echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
      export NSS_WRAPPER_PASSWD=/tmp/passwd
      export LD_PRELOAD=libnss_wrapper.so
      export NSS_WRAPPER_GROUP=/etc/group
      fi
      agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile ${AGENT_LOG_FILE} -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS} -logLevel ${AGENT_LOG_LEVEL}
    State:          Running
      Started:      Thu, 02 Nov 2023 10:25:03 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  512Mi
    Requests:
      cpu:      200m
      memory:   512Mi
    Readiness:  exec [/opt/scripts/readinessprobe] delay=5s timeout=1s period=10s #success=1 #failure=40
    Environment:
      AGENT_LOG_FILE:                     /var/log/mongodb-mms-automation/automation-agent.log
      AGENT_LOG_LEVEL:                    INFO
      AGENT_MAX_LOG_FILE_DURATION_HOURS:  24
      AGENT_STATUS_FILEPATH:              /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
      AUTOMATION_CONFIG_MAP:              mongo-rs-config
      HEADLESS_AGENT:                     true
      POD_NAMESPACE:                      default (v1:metadata.namespace)
    Mounts:
      /data from data-volume (rw)
      /opt/scripts from agent-scripts (rw)
      /tmp from tmp (rw)
      /var/lib/automation/config from automation-config (ro)
      /var/lib/mongodb-mms-automation/authentication from mongo-rs-keyfile (rw)
      /var/log/mongodb-mms-automation from logs-volume (rw)
      /var/log/mongodb-mms-automation/healthstatus from healthstatus (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tpff9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-volume-mongo-rs-0
    ReadOnly:   false
  logs-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  logs-volume-mongo-rs-0
    ReadOnly:   false
  agent-scripts:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  automation-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  mongo-rs-config
    Optional:    false
  healthstatus:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  hooks:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  mongo-rs-keyfile:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-tpff9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  118s               default-scheduler  Successfully assigned default/mongo-rs-0 to docker-desktop
  Normal   Pulling    118s               kubelet            Pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8"
  Normal   Pulled     116s               kubelet            Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8" in 1.590324293s (1.590365792s including waiting)
  Normal   Created    116s               kubelet            Created container mongod-posthook
  Normal   Started    116s               kubelet            Started container mongod-posthook
  Normal   Pulling    115s               kubelet            Pulling image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17-arm64"
  Normal   Pulled     114s               kubelet            Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17-arm64" in 1.499687834s (1.499722667s including waiting)
  Normal   Created    114s               kubelet            Created container mongodb-agent-readinessprobe
  Normal   Started    114s               kubelet            Started container mongodb-agent-readinessprobe
  Normal   Pulling    113s               kubelet            Pulling image "docker.io/mongo:6.0.11"
  Normal   Pulled     98s                kubelet            Successfully pulled image "docker.io/mongo:6.0.11" in 15.391279923s (15.391326466s including waiting)
  Normal   Created    97s                kubelet            Created container mongod
  Normal   Started    97s                kubelet            Started container mongod
  Normal   Pulling    97s                kubelet            Pulling image "quay.io/mongodb/mongodb-agent:12.0.25.7724-1"
  Normal   Pulled     96s                kubelet            Successfully pulled image "quay.io/mongodb/mongodb-agent:12.0.25.7724-1" in 1.658521668s (1.658647918s including waiting)
  Normal   Created    95s                kubelet            Created container mongodb-agent
  Normal   Started    95s                kubelet            Started container mongodb-agent
  Warning  Unhealthy  78s (x2 over 87s)  kubelet            Readiness probe failed: panic: open /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json: no such file or directory

goroutine 1 [running]:
main.main()
           /workspace/cmd/readiness/main.go:226 +0x160
  Warning  Unhealthy  37s                kubelet  Readiness probe failed: command "/opt/scripts/readinessprobe" timed out
  Warning  Unhealthy  18s (x6 over 68s)  kubelet  Readiness probe failed:

yaml definitions mongo rs:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2023-11-02T09:24:40Z"
  generateName: mongo-rs-
  labels:
    app: mongo-rs-svc
    controller-revision-hash: mongo-rs-7c746d48d7
    statefulset.kubernetes.io/pod-name: mongo-rs-0
  name: mongo-rs-0
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: mongo-rs
    uid: 13a90dd8-ebf4-454b-8611-0f62867324a9
  resourceVersion: "181335"
  uid: a2df9712-5e34-4a1d-9156-ea1d906486e9
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - mongo-replicaset
        topologyKey: kubernetes.io/hostname
  containers:
  - args:
    - ""
    command:
    - /bin/sh
    - -c
    - |2+

      #run post-start hook to handle version changes
      /hooks/version-upgrade

      # wait for config and keyfile to be created by the agent
       while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

      # start mongod with this configuration
      exec mongod -f /data/automation-mongod.conf;

    env:
    - name: AGENT_STATUS_FILEPATH
      value: /healthstatus/agent-health-status.json
    image: docker.io/mongo:6.0.11
    imagePullPolicy: IfNotPresent
    name: mongod
    resources:
      limits:
        cpu: 200m
        memory: 214748364800m
      requests:
        cpu: 200m
        memory: 214748364800m
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /data
      name: data-volume
    - mountPath: /healthstatus
      name: healthstatus
    - mountPath: /hooks
      name: hooks
    - mountPath: /var/log/mongodb-mms-automation
      name: logs-volume
    - mountPath: /var/lib/mongodb-mms-automation/authentication
      name: mongo-rs-keyfile
    - mountPath: /tmp
      name: tmp
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-tpff9
      readOnly: true
  - command:
    - /bin/bash
    - -c
    - |-
      current_uid=$(id -u)
      AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
      declare -r current_uid
      if ! grep -q "${current_uid}" /etc/passwd ; then
      sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
      echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
      export NSS_WRAPPER_PASSWD=/tmp/passwd
      export LD_PRELOAD=libnss_wrapper.so
      export NSS_WRAPPER_GROUP=/etc/group
      fi
      agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile ${AGENT_LOG_FILE} -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS} -logLevel ${AGENT_LOG_LEVEL}
    env:
    - name: AGENT_LOG_FILE
      value: /var/log/mongodb-mms-automation/automation-agent.log
    - name: AGENT_LOG_LEVEL
      value: INFO
    - name: AGENT_MAX_LOG_FILE_DURATION_HOURS
      value: "24"
    - name: AGENT_STATUS_FILEPATH
      value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
    - name: AUTOMATION_CONFIG_MAP
      value: mongo-rs-config
    - name: HEADLESS_AGENT
      value: "true"
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    image: quay.io/mongodb/mongodb-agent:12.0.25.7724-1
    imagePullPolicy: Always
    name: mongodb-agent
    readinessProbe:
      exec:
        command:
        - /opt/scripts/readinessprobe
      failureThreshold: 40
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: 200m
        memory: 512Mi
      requests:
        cpu: 200m
        memory: 512Mi
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/scripts
      name: agent-scripts
    - mountPath: /var/lib/automation/config
      name: automation-config
      readOnly: true
    - mountPath: /data
      name: data-volume
    - mountPath: /var/log/mongodb-mms-automation/healthstatus
      name: healthstatus
    - mountPath: /var/log/mongodb-mms-automation
      name: logs-volume
    - mountPath: /var/lib/mongodb-mms-automation/authentication
      name: mongo-rs-keyfile
    - mountPath: /tmp
      name: tmp
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-tpff9
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: mongo-rs-0
  initContainers:
  - command:
    - cp
    - version-upgrade-hook
    - /hooks/version-upgrade
    image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
    imagePullPolicy: Always
    name: mongod-posthook
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /hooks
      name: hooks
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-tpff9
      readOnly: true
  - command:
    - cp
    - /probes/readinessprobe
    - /opt/scripts/readinessprobe
    image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17-arm64
    imagePullPolicy: Always
    name: mongodb-agent-readinessprobe
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/scripts
      name: agent-scripts
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-tpff9
      readOnly: true
  nodeName: docker-desktop
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 2000
  serviceAccount: mongodb-database
  serviceAccountName: mongodb-database
  subdomain: mongo-rs-svc
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: data-volume
    persistentVolumeClaim:
      claimName: data-volume-mongo-rs-0
  - name: logs-volume
    persistentVolumeClaim:
      claimName: logs-volume-mongo-rs-0
  - emptyDir: {}
    name: agent-scripts
  - name: automation-config
    secret:
      defaultMode: 416
      secretName: mongo-rs-config
  - emptyDir: {}
    name: healthstatus
  - emptyDir: {}
    name: hooks
  - emptyDir: {}
    name: mongo-rs-keyfile
  - emptyDir: {}
    name: tmp
  - name: kube-api-access-tpff9
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-11-02T09:24:45Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-11-02T09:24:40Z"
    message: 'containers with unready status: [mongod mongodb-agent]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-11-02T09:24:40Z"
    message: 'containers with unready status: [mongod mongodb-agent]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-11-02T09:24:40Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: docker.io/mongo:6.0.11
    imageID: ""
    lastState: {}
    name: mongod
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
  - image: quay.io/mongodb/mongodb-agent:12.0.25.7724-1
    imageID: ""
    lastState: {}
    name: mongodb-agent
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
  hostIP: 192.168.65.3
  initContainerStatuses:
  - containerID: docker://41885b3e6f3d6318855be29896e7704bacb0e8a49d2cde2e70989b370b730cfc
    image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
    imageID: docker-pullable://quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:5ebea828235618936f059a89048098fca662cd665a0589ea25f5cbee2676a2ed
    lastState: {}
    name: mongod-posthook
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://41885b3e6f3d6318855be29896e7704bacb0e8a49d2cde2e70989b370b730cfc
        exitCode: 0
        finishedAt: "2023-11-02T09:24:42Z"
        reason: Completed
        startedAt: "2023-11-02T09:24:42Z"
  - containerID: docker://b209116d6165333f5a3a1e743938e181385e8c4e64d93c41d677dfb9117b95d5
    image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17-arm64
    imageID: docker-pullable://quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:43b2bba2b7d7ab157d707f365bfaf7a8eb080969a025af09be99c3885e583556
    lastState: {}
    name: mongodb-agent-readinessprobe
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://b209116d6165333f5a3a1e743938e181385e8c4e64d93c41d677dfb9117b95d5
        exitCode: 0
        finishedAt: "2023-11-02T09:24:44Z"
        reason: Completed
        startedAt: "2023-11-02T09:24:44Z"
  phase: Pending
  podIP: 10.1.1.177
  podIPs:
  - ip: 10.1.1.177
  qosClass: Burstable
  startTime: "2023-11-02T09:24:40Z"

PVs:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .Values.mongo_pv.data.name}}
  labels:
    type: local
spec:
  capacity:
    storage: {{ .Values.mongo_pv.data.storage}}
  persistentVolumeReclaimPolicy: {{ .Values.mongo_pv.data.reclaimPolicy}} #Delete
  accessModes:
    - {{ .Values.mongo_pv.data.accessMode}}
  volumeMode: Filesystem
  storageClassName: {{ .Values.storage_class.data.name}}
  local:
    path: {{ .Values.mongo_pv.data.path }}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                # - docker-desktop
                - k3d-test-server-0
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name:  {{ .Values.mongo_pv.logs.name}}
  labels:
    type: local
spec:
  capacity:
    storage: {{ .Values.mongo_pv.logs.storage}}
  persistentVolumeReclaimPolicy: {{ .Values.mongo_pv.logs.reclaimPolicy}}
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  storageClassName: {{ .Values.storage_class.logs.name}}
  local:
    path: {{ .Values.mongo_pv.logs.path}}

  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                # - docker-desktop
                - k3d-test-server-0

values:

namespace: default
# mongo_pv persistent volumes are only for local cluster deployment
mongo_pv:
  data: 
    name: mongo-data-pv
    storage: 4Gi
    reclaimPolicy: Retain
    accessMode: ReadWriteOnce
    path: /Volumes/ProjectsSSD/k8s_local_volumes/m-data
  logs: 
    name: mongo-logs-pv
    storage: 4Gi
    reclaimPolicy: Retain
    accessMode: ReadWriteOnce
    path: /Volumes/ProjectsSSD/k8s_local_volumes/m-logs

# sc
node_label_key: server-type
node_label_value: master
storage_class:
  data: 
    name: mongo-sc-data
    provisioner: kubernetes.io/no-provisioner
    reclaimPolicy: Retain
    volumeBindingMode: WaitForFirstConsumer
    allowVolumeExpansion: true

  logs: 
    name: mongo-sc-logs
    provisioner: kubernetes.io/no-provisioner
    reclaimPolicy: Retain
    volumeBindingMode: WaitForFirstConsumer
    allowVolumeExpansion: true
# vpc
volume_claim_templates:
  data:
    name: data-volume
    storage: 4Gi
    access_mode: ReadWriteOnce
  logs: 
    name: logs-volume
    storage: 4Gi
    access_mode: ReadWriteOnce

# pv
persistent_volumes:
  data:
    name: data-mongo-pv

  data1:
    name: data1-mongo-pv  

  logs:
    name: logs-mongo-pv
  logs1:
    name: logs1-mongo-pv

# secret
secret:
  name: mongo-secret
  password: password

# replica set
replica_set:
  replicas: 1
  name: mongo-rs
  # mongo_version: '7.0.2' # image not found
  mongo_version: '6.0.11'
  # mongo_version: '6.0.6'
  # mongo_version: '5.0.5'
  db_name: fixit
  admin_user:
   name: admin-user
   role_1: clusterAdmin
   role_2: userAdminAnyDatabase
   role_3: readWrite
   scramCredentialsSecretName: my-scram-mg-fixit 

resources:
  mongod:
    limits:
      cpu: "0.2"
      memory: 0.2Gi
    requests:
      cpu: "0.2"
      memory: 0.2Gi
  mongodb_agent:
    limits:
      cpu: "0.2"
      memory: 0.5Gi
    requests: 
      cpu: "0.2"
      memory: 0.5Gi      
vinnytwice commented 8 months ago
vinnytwice commented 8 months ago

operator pod logs

2023-11-01T14:37:26.107Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:26.590Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:37:26.592Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:26.592Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:26.592Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:26.593Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:37:26.595Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:26.595Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:26.595Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:37:36.688Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:36.690Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:36.691Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:36.698Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:36.699Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:36.788Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:37:36.789Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:37.592Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:37:37.594Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:37.595Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:37.595Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:37.595Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:37:37.680Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:37.681Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:37.681Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:37:47.707Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:47.708Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:47.708Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:47.713Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:47.714Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:47.737Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:37:47.780Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:48.488Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:37:48.489Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:48.490Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:48.490Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:48.490Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:37:48.491Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:48.491Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:48.491Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:37:58.503Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.505Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.506Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.509Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.510Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.521Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:37:58.522Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.990Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:37:58.991Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.991Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.992Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.992Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:37:58.993Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.993Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:37:58.993Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:38:09.083Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.085Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.086Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.092Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.093Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.179Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:38:09.181Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.881Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:38:09.882Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.882Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.882Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.883Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:38:09.884Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.885Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:09.885Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:38:19.897Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:19.897Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:19.898Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:19.901Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:19.901Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:19.916Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:38:20.179Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:21.184Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:38:21.186Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:21.186Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:21.186Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:21.186Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:38:21.187Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:21.188Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:21.188Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:38:31.200Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.201Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.201Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.204Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.204Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.211Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:38:31.212Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.596Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:38:31.678Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.678Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.678Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.678Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:38:31.680Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.680Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:31.680Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:38:41.691Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:41.691Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:41.692Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:41.694Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:41.695Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:41.704Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:38:41.705Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:42.882Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:38:42.886Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:42.887Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:42.887Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:42.887Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:38:42.892Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:42.893Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:42.893Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:38:53.012Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.013Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.013Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.017Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.018Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.033Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:38:53.077Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.783Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:38:53.784Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.784Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.785Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.785Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:38:53.786Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.786Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:38:53.786Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:39:03.804Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:03.805Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:03.806Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:03.810Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:03.811Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:03.826Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:39:03.827Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:04.392Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:39:04.394Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:04.394Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:04.394Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:04.394Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:39:04.476Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:04.476Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:04.476Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:39:14.488Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:14.489Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:14.490Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:14.493Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:14.494Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:14.503Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:39:14.504Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:15.077Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:39:15.078Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:15.078Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:15.078Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:15.078Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:39:15.080Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:15.080Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:15.080Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:39:25.096Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:25.097Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:25.097Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:25.100Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:25.101Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:25.124Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:39:25.126Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:26.088Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:39:26.090Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:26.090Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:26.091Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:26.091Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:39:26.094Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:26.094Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:26.094Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:39:36.205Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.205Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.206Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.209Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.209Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.220Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:39:36.222Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.686Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:39:36.687Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.687Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.687Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.687Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:39:36.689Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.689Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:36.689Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:39:46.704Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:46.706Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:46.707Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:46.714Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:46.715Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:46.788Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:39:46.790Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:47.491Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:39:47.492Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:47.492Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:47.493Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:47.493Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:39:47.494Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:47.495Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:47.495Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:39:57.582Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:57.583Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:57.584Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:57.590Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:57.591Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:57.615Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:39:57.670Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:58.384Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:39:58.385Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:58.385Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:58.385Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:58.385Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:39:58.386Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:58.386Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:39:58.386Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:40:08.402Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:08.404Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:08.408Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:08.469Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:08.470Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:08.492Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:40:08.494Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:09.178Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:40:09.180Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:09.180Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:09.180Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:09.180Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:40:09.183Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:09.187Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:09.269Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:40:19.308Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:19.310Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:19.311Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:19.373Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:19.375Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:19.470Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:40:19.472Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:20.176Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:40:20.177Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:20.178Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:20.178Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:20.178Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:40:20.179Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:20.180Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:20.180Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:40:30.196Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.197Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.198Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.201Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.202Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.220Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:40:30.221Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.974Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:40:30.975Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.975Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.975Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.975Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:40:30.977Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.977Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:30.977Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:40:40.993Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:40.995Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:40.997Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:41.003Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:41.005Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:41.083Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:40:41.085Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:41.775Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:40:41.776Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:41.776Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:41.776Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:41.776Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:40:41.778Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:41.778Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:41.778Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:40:51.791Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:51.792Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:51.793Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:51.797Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:51.799Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:51.816Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:40:51.873Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:52.878Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:40:52.881Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:52.881Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:52.881Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:52.881Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:40:52.884Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:52.885Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:40:52.885Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:41:02.908Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:02.909Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:02.909Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:02.913Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:02.914Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:02.942Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:41:02.943Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:03.374Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:41:03.375Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:03.375Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:03.375Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:03.375Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:41:03.376Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:03.376Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:03.377Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:41:13.399Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:13.400Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:13.401Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:13.406Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:13.407Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:13.473Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:41:13.474Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:14.181Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:41:14.184Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:14.184Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:14.185Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:14.185Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:41:14.259Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:14.270Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:14.271Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:41:24.289Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:24.290Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:24.291Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:24.296Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:24.298Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:24.380Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:41:24.381Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:25.274Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:41:25.275Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:25.275Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:25.275Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:25.275Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:41:25.277Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:25.277Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:25.277Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:41:35.295Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:35.297Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:35.299Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:35.305Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:35.306Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:35.393Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:41:35.394Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:36.387Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:41:36.471Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:36.471Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:36.471Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:36.471Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:41:36.474Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:36.474Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:36.475Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:41:46.493Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:46.494Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:46.495Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:46.498Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:46.499Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:46.513Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:41:46.515Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:47.072Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:41:47.073Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:47.073Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:47.073Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:47.073Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:41:47.075Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:47.075Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:47.075Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:41:57.080Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:57.081Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:57.082Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:57.085Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:57.088Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:57.164Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:41:57.167Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:58.776Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:41:58.778Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:58.778Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:58.779Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:58.779Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:41:58.781Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:58.781Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:41:58.782Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:42:08.888Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:08.888Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:08.889Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:08.893Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:08.895Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:08.917Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:42:08.965Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:10.076Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:42:10.080Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:10.080Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:10.080Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:10.081Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:42:10.170Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:10.172Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:10.173Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:42:20.272Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.274Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.275Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.280Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.281Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.375Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:42:20.377Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.881Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:42:20.882Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.964Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.964Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.964Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:42:20.966Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.966Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:20.966Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:42:30.979Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:30.979Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:30.980Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:30.982Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:30.982Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:30.991Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:42:30.991Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:31.783Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:42:31.785Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:31.786Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:31.786Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:31.868Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:42:31.871Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:31.871Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:31.871Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:42:41.886Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:41.886Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:41.887Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:41.888Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:41.889Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:41.898Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:42:41.899Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:42.670Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:42:42.671Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:42.671Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:42.671Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:42.671Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:42:42.672Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:42.672Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:42.672Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:42:52.680Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:52.680Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:52.680Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:52.681Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:52.681Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:52.685Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:42:52.685Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:53.082Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:42:53.083Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:53.083Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:53.083Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:53.083Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:42:53.085Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:53.085Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:42:53.085Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:43:03.169Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.170Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.170Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.172Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.172Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.182Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:43:03.183Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.976Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:43:03.977Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.977Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.977Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.977Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:43:03.978Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.978Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:03.979Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:43:14.000Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:14.003Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:14.006Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:14.072Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:14.073Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:14.094Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:43:14.175Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:15.084Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:43:15.085Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:15.166Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:15.167Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:15.167Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:43:15.170Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:15.171Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:15.171Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:43:25.189Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:25.190Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:25.191Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:25.196Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:25.197Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:25.267Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:43:25.269Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:26.081Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:43:26.082Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:26.082Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:26.082Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:26.083Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:43:26.084Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:26.084Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:26.084Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:43:36.189Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:36.191Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:36.191Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:36.196Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:36.197Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:36.211Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:43:36.267Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:37.178Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:43:37.180Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:37.180Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:37.180Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:37.180Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:43:37.181Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:37.181Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:37.181Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:43:47.279Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.280Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.285Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.289Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.290Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.384Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:43:47.386Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.974Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:43:47.975Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.975Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.976Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.976Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:43:47.978Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.978Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:47.978Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:43:58.001Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:58.001Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:58.002Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:58.005Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:58.006Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:58.020Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:43:58.022Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:59.669Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:43:59.676Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:59.677Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:59.677Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:59.677Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:43:59.681Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:59.682Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:43:59.682Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:44:09.788Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:09.789Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:09.790Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:09.793Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:09.794Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:09.806Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:44:09.808Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:10.678Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:44:10.681Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:10.682Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:10.765Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:10.765Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:44:10.767Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:10.767Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:10.767Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:44:20.784Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:20.784Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:20.785Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:20.789Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:20.790Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:20.813Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:44:20.864Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:21.382Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:44:21.466Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:21.466Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:21.466Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:21.467Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:44:21.470Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:21.470Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:21.470Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:44:31.488Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:31.488Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:31.489Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:31.491Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:31.492Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:31.507Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:44:31.508Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:32.573Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:44:32.578Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:32.578Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:32.578Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:32.579Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:44:32.663Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:32.664Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:32.664Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:44:42.688Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:42.689Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:42.690Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:42.695Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:42.696Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:42.715Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:44:42.766Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:43.474Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:44:43.476Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:43.476Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:43.476Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:43.476Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:44:43.478Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:43.478Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:43.478Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:44:53.488Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:53.488Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:53.489Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:53.490Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:53.490Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:53.501Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:44:53.502Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:54.066Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:44:54.162Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:54.163Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:54.163Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:54.163Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:44:54.167Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:54.168Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:44:54.168Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:45:04.265Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.265Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.265Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.267Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.268Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.276Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:45:04.277Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.771Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:45:04.772Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.773Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.773Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.773Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:45:04.774Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.774Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:04.774Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:45:14.782Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:14.782Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:14.782Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:14.783Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:14.783Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:14.787Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:45:14.787Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:15.263Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:45:15.265Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:15.265Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:15.265Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:15.265Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:45:15.267Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:15.268Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:15.268Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:45:25.279Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:25.280Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:25.280Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:25.283Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:25.284Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:25.294Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:45:25.295Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:26.072Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:45:26.074Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:26.074Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:26.075Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:26.075Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:45:26.077Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:26.077Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:26.078Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:45:36.183Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.183Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.184Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.187Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.187Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.199Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:45:36.200Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.762Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:45:36.763Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.763Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.763Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.763Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:45:36.764Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.765Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:36.765Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:45:46.777Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:46.778Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:46.779Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:46.783Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:46.784Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:46.800Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:45:46.801Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:47.770Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:45:47.772Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:47.773Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:47.773Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:47.773Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:45:47.775Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:47.775Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:47.776Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:45:57.872Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:57.873Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:57.877Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:58.162Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:58.163Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:58.258Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:45:58.258Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:58.674Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:45:58.675Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:58.676Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:58.676Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:58.676Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:45:58.678Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:58.759Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:45:58.759Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:46:08.775Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:08.776Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:08.776Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:08.779Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:08.779Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:08.790Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:46:08.791Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:09.472Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:46:09.474Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:09.474Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:09.474Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:09.474Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:46:09.547Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:09.547Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:09.547Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:46:19.580Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:19.583Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:19.584Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:19.661Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:19.662Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:19.680Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:46:19.682Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:20.175Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:46:20.176Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:20.176Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:20.176Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:20.176Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:46:20.258Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:20.259Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:20.259Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:46:30.273Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.274Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.275Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.278Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.279Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.298Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:46:30.299Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.774Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:46:30.775Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.775Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.775Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.775Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:46:30.859Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.859Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:30.859Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:46:40.871Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:40.872Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:40.873Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:40.877Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:40.878Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:40.891Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:46:40.957Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:41.773Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:46:41.775Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:41.775Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:41.775Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:41.775Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:46:41.858Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:41.859Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:41.859Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:46:51.872Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:51.873Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:51.873Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:51.876Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:51.877Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:51.887Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:46:51.889Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:52.757Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:46:52.759Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:52.759Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:52.759Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:52.759Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:46:52.761Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:52.761Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:46:52.761Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:47:02.774Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:02.775Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:02.775Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:02.777Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:02.778Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:02.790Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:47:02.792Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:03.376Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:47:03.458Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:03.458Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:03.458Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:03.458Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:47:03.460Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:03.461Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:03.461Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:47:13.473Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:13.474Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:13.474Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:13.476Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:13.476Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:13.484Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:47:13.485Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:14.360Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:47:14.362Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:14.363Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:14.363Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:14.363Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:47:14.366Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:14.366Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:14.366Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:47:24.381Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.382Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.383Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.387Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.388Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.397Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:47:24.398Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.866Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:47:24.867Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.867Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.867Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.867Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:47:24.954Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.955Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:24.955Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:47:35.057Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.057Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.058Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.064Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.069Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.082Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:47:35.084Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.669Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:47:35.671Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.671Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.671Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.671Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:47:35.753Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.753Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:35.754Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:47:45.768Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:45.769Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:45.770Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:45.774Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:45.775Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:45.791Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:47:45.792Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:46.757Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:47:46.758Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:46.758Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:46.758Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:46.758Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:47:46.760Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:46.760Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:46.760Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:47:56.770Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:56.771Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:56.772Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:56.776Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:56.777Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:56.792Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:47:56.794Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:58.056Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:47:58.057Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:58.058Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:58.058Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:58.058Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:47:58.060Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:58.060Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:47:58.060Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:48:08.074Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.076Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.077Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.082Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.084Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.152Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:48:08.154Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.564Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:48:08.564Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.565Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.565Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.565Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:48:08.566Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.566Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:08.566Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:48:18.576Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:18.576Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:18.577Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:18.580Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:18.580Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:18.592Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:48:18.593Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:19.841Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:48:19.854Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:19.854Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:19.855Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:19.855Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:48:19.858Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:19.859Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:19.859Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:48:29.891Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:29.892Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:29.893Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:29.897Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:29.898Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:29.951Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:48:29.952Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:30.460Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:48:30.462Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:30.462Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:30.462Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:30.462Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:48:30.464Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:30.465Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:30.465Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:48:40.558Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:40.560Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:40.561Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:40.566Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:40.567Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:40.663Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:48:40.665Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:41.357Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:48:41.359Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:41.360Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:41.360Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:41.360Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:48:41.363Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:41.364Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:41.364Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:48:51.385Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:51.386Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:51.387Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:51.392Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:51.393Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:51.405Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:48:51.406Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:51.972Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:48:52.053Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:52.054Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:52.054Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:52.054Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:48:52.057Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:52.057Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:48:52.057Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:49:02.071Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:02.071Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:02.072Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:02.075Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:02.076Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:02.092Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:49:02.093Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:03.255Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:49:03.352Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:03.353Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:03.353Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:03.353Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:49:03.461Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:03.551Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:03.551Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:49:13.564Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:13.564Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:13.564Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:13.566Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:13.566Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:13.574Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:49:13.575Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:14.163Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:49:14.165Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:14.165Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:14.166Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:14.166Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:49:14.168Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:14.169Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:14.169Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:49:24.268Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:24.269Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:24.270Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:24.274Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:24.275Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:24.290Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:49:24.292Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:25.050Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:49:25.061Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:25.063Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:25.065Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:25.149Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:49:25.152Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:25.152Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:25.153Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:49:35.174Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:35.174Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:35.174Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:35.176Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:35.176Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:35.181Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:49:35.182Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:36.049Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:49:36.052Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:36.053Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:36.053Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:36.053Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:49:36.055Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:36.056Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:36.056Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:49:46.075Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.075Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.076Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.078Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.079Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.086Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:49:46.087Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.760Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:49:46.761Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.762Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.762Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.762Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:49:46.764Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.764Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:46.764Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
2023-11-01T14:49:56.951Z        INFO    controllers/replica_set_controller.go:134       Reconciling MongoDB     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:56.952Z        DEBUG   controllers/replica_set_controller.go:136       Validating MongoDB.Spec {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:56.952Z        DEBUG   controllers/replica_set_controller.go:146       Ensuring the service exists     {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:56.955Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:56.956Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:56.963Z        INFO    controllers/replica_set_controller.go:472       Create/Update operation succeeded       {"ReplicaSet": "default/mongo-rs", "operation": "updated"}
2023-11-01T14:49:56.964Z        INFO    controllers/replica_set_controller.go:364       Creating/Updating AutomationConfig      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:57.448Z        DEBUG   scram/scram.go:101      Credentials have not changed, using credentials stored in: secret/my-scram-mg-fixit-scram-credentials
2023-11-01T14:49:57.454Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:57.555Z        DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:57.556Z        DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongo-rs-0:27017]      {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:57.556Z        DEBUG   controllers/replica_set_controller.go:539       AutomationConfigMembersThisReconciliation       {"mdb.AutomationConfigMembersThisReconciliation()": 1}
2023-11-01T14:49:57.651Z        DEBUG   controllers/replica_set_controller.go:387       Waiting for agents to reach version 1   {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:57.652Z        DEBUG   agent/agent_readiness.go:106    The Pod 'mongo-rs-0' doesn't have annotation 'agent.mongodb.com/version' yet    {"ReplicaSet": "default/mongo-rs"}
2023-11-01T14:49:57.652Z        INFO    controllers/mongodb_status_options.go:110       ReplicaSet is not yet ready, retrying in 10 seconds
vinnytwice commented 8 months ago

The problem is with the mongo-agent which doesn't have an official arm64 image yet ( https://quay.io/repository/mongodb/mongodb-agent?tab=tags) while operator, version-upgrade-post-start-hook and readinessprobe do, so when installing mongo via the operator (the operator points to the quay.io images) the mongo-agent that gets installed not compatible with the M1 and mongo pods hang as READY 1/2 with Readiness probe failed: command "/opt/scripts/readinessprobe" timed out. Dough the official arm64 image for the agent is not available yet, the binaries are available and thankfully they've been built by @mohsinonxrm and made available on DockerHub see here https://github.com/mongodb/mongodb-kubernetes-operator/issues/299#issuecomment-1666717878. So all it's needed is to override the operator chart values to use those images instead of the official at quay.io as:

operator:
  name: mongodb-kubernetes-operator
  operatorImageName: mongodb-kubernetes-operator
  deploymentName: mongodb-kubernetes-operator
  version: 0.8.1
  resources:
    limits:
      cpu: 200m 
      memory: 200Mi 
    requests:
      cpu: 200m 
      memory: 200Mi 
agent:
  name: mongodb-agent
  version: 12.0.24.7719-1
versionUpgradeHook:
  name: mongodb-kubernetes-operator-version-upgrade-post-start-hook
  version: 1.0.7
readinessProbe:
  name: mongodb-kubernetes-readiness
  version: 1.0.15

registry:
  agent: docker.io/mohsinonxrm #quay.io/mongodb
  versionUpgradeHook: docker.io/mohsinonxrm #quay.io/mongodb
  readinessProbe: docker.io/mohsinonxrm #quay.io/mongodb
  operator: docker.io/mohsinonxrm #quay.io/mongodb

Hopefully official arm64 image for the mongo-agent will be released soon so to use operator 0.8.3.

Julien-Ben commented 8 months ago

@vinnytwice As the ARM images for the agent were recently released, can you check if you still have the same issue ?

github-actions[bot] commented 6 months ago

This issue is being marked stale because it has been open for 60 days with no activity. Please comment if this issue is still affecting you. If there is no change, this issue will be closed in 30 days.

slaskawi commented 5 months ago

Let me assign this to @wtrocki

nammn commented 4 months ago

@vinnytwice you are pointing to the wrong agent image, the correct one is: https://quay.io/repository/mongodb/mongodb-agent-ubi?tab=tags

vinnytwice commented 4 months ago

@nammn Hi and thanks for pointing that out. So my values should be like the following correct?

mongodb-community-operator:
  operator:
    name: mongodb-kubernetes-operator
    operatorImageName: mongodb-kubernetes-operator
    deploymentName: mongodb-kubernetes-operator
    # version: 0.8.1
    version: 0.8.3-arm64
    resources:
      limits:
        cpu: 200m #1100m
        memory: 200Mi #1Gi
      requests:
        cpu: 200m #500m
        memory: 200Mi #500Mi
  agent:
    name: mongodb-agent
    # version: 12.0.24.7719-1
    version: 12.0.25.7724-1 # should work
    # version: 107.0.0.8465-1 # latest
  versionUpgradeHook:
    name: mongodb-kubernetes-operator-version-upgrade-post-start-hook
    #version: 1.0.7
    version: 1.0.8-arm64 #latest
  readinessProbe:
    name: mongodb-kubernetes-readiness
    # version: 1.0.15
    # version: 1.0.17-b20240301T000000Z-arm64
    version: 1.0.17-arm64 #latest
  # mongodb:
  #   name: zcube/bitnami-compat-mongodb #mongo
  #   repo: docker.io

  # registry:
  #   # unofficial images, working with ARM chips
  #   agent: docker.io/mohsinonxrm
  #   versionUpgradeHook: docker.io/mohsinonxrm
  #   readinessProbe: docker.io/mohsinonxrm
  #   operator: docker.io/mohsinonxrm
  #   pullPolicy: Always
  registry:
    # official images, working with ARM chips ?
    agent: quay.io/repository/mongodb/mongodb-agent-ubi # for ubuntu machines 
    versionUpgradeHook: quay.io/repository/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook
    readinessProbe: quay.io/repository/mongodb/mongodb-kubernetes-readiness
    operator: quay.io/repository/mongodb/mongodb-kubernetes-operator 
    pullPolicy: Always
aleksandrov commented 3 months ago

I tried fresh images on my raspberry pi cluster:

operator:
  name: mongodb-kubernetes-operator
  version: 0.9.0-arm64
  deploymentName: mongodb-kubernetes-operator
  resources:
    limits:
      cpu: 200m
      memory: 200Mi
    requests:
      cpu: 200m
      memory: 200Mi

agent:
  name: mongodb-agent-ubi
  version: 12.0.28.7763-1-arm64
versionUpgradeHook:
  name: mongodb-kubernetes-operator-version-upgrade-post-start-hook
  version: 1.0.8-arm64
readinessProbe:
  name: mongodb-kubernetes-readinessprobe
  version: 1.0.17-arm64

still no luck:

$ kubectl -n mongodb logs pod/home-mongodb-0 -c mongodb-agent
cat: /mongodb-automation/agent-api-key/agentApiKey: No such file or directory
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Warning  Unhealthy  1s (x2 over 2s)   kubelet            Readiness probe failed: panic: open /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json: no such file or directory

goroutine 1 [running]:
main.main()
  /workspace/cmd/readiness/main.go:226 +0x160
github-actions[bot] commented 1 month ago

This issue is being marked stale because it has been open for 60 days with no activity. Please comment if this issue is still affecting you. If there is no change, this issue will be closed in 30 days.

abasu0713 commented 4 weeks ago

I tried fresh images on my raspberry pi cluster:

operator:
  name: mongodb-kubernetes-operator
  version: 0.9.0-arm64
  deploymentName: mongodb-kubernetes-operator
  resources:
    limits:
      cpu: 200m
      memory: 200Mi
    requests:
      cpu: 200m
      memory: 200Mi

agent:
  name: mongodb-agent-ubi
  version: 12.0.28.7763-1-arm64
versionUpgradeHook:
  name: mongodb-kubernetes-operator-version-upgrade-post-start-hook
  version: 1.0.8-arm64
readinessProbe:
  name: mongodb-kubernetes-readinessprobe
  version: 1.0.17-arm64

still no luck:

$ kubectl -n mongodb logs pod/home-mongodb-0 -c mongodb-agent
cat: /mongodb-automation/agent-api-key/agentApiKey: No such file or directory
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Warning  Unhealthy  1s (x2 over 2s)   kubelet            Readiness probe failed: panic: open /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json: no such file or directory

goroutine 1 [running]:
main.main()
  /workspace/cmd/readiness/main.go:226 +0x160

@aleksandrov I got it working on an Orange Pi. Which means you should be able to run it on a RPI. Below is my operator config:

## Reference to one or more secrets to be used when pulling images
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
imagePullSecrets: []
# - name: "image-pull-secret"
## Operator
operator:
  # Name that will be assigned to most of internal Kubernetes objects like
  # Deployment, ServiceAccount, Role etc.
  name: mongodb-kubernetes-operator

  # Name of the operator image
  operatorImageName: mongodb-kubernetes-operator

  # Name of the deployment of the operator pod
  deploymentName: mongodb-kubernetes-operator

  # Version of mongodb-kubernetes-operator
  # version: 0.8.1
  version: 0.9.0

  # Uncomment this line to watch all namespaces
  # watchNamespace: "*"

  # Resources allocated to Operator Pod
  resources:
    limits:
      cpu: 750m
      memory: 500Mi
    requests:
      cpu: 300m
      memory: 200Mi

  # replicas deployed for the operator pod. Running 1 is optimal and suggested.
  replicas: 1

  # Additional environment variables
  extraEnvs: []
  # environment:
  # - name: CLUSTER_DOMAIN
  #   value: my-cluster.domain

  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 2000

  securityContext: {}

## Operator's database
database:
  name: mongodb-database
  # set this to the namespace where you would like
  # to deploy the MongoDB database,
  # Note if the database namespace is not same
  # as the operator namespace,
  # make sure to set "watchNamespace" to "*"
  # to ensure that the operator has the
  # permission to reconcile resources in other namespaces
  # namespace: mongodb-database

agent:
  name: mongodb-agent-ubi
  version: 107.0.6.8587-1-arm64
versionUpgradeHook:
  name: mongodb-kubernetes-operator-version-upgrade-post-start-hook
  version: 1.0.8
readinessProbe:
  name: mongodb-kubernetes-readinessprobe
  version: 1.0.17
mongodb:
  name: mongo
  repo: docker.io

registry:
  # agent: docker.io/mohsinonxrm
  # versionUpgradeHook: docker.io/mohsinonxrm
  # readinessProbe: docker.io/mohsinonxrm
  # operator: docker.io/mohsinonxrm
  agent: quay.io/mongodb
  versionUpgradeHook: quay.io/mongodb
  readinessProbe: quay.io/mongodb
  operator: quay.io/mongodb
  pullPolicy: Always

# Set to false if CRDs have been installed already. The CRDs can be installed
# manually from the code repo: github.com/mongodb/mongodb-kubernetes-operator or
# using the `community-operator-crds` Helm chart.
community-operator-crds:
  enabled: true