search

mongodb / mongodb-kubernetes-operator

MongoDB Community Kubernetes Operator
Other
1.2k stars 485 forks source link

failed to create containerd task: failed to create shim #1557

Closed zeitler closed 6 days ago

zeitler commented 3 months ago

What did you do to encounter the bug? Clean install and it simply dont start. Already installed in another cluster, never had issues

What did you expect? The statefulSet gets up and running

What happened instead? First member fails to start

Screenshots

13s                   Normal    Pulling     Pod/mongodb-0   Pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8"
11s                   Normal    Pulled      Pod/mongodb-0   Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8" in 1.845683651s
11s                   Normal    Created     Pod/mongodb-0   Created container mongod-posthook
11s                   Normal    Started     Pod/mongodb-0   Started container mongod-posthook
11s                   Normal    Pulling     Pod/mongodb-0   Pulling image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17"
10s                   Normal    Pulled      Pod/mongodb-0   Successfully pulled image "quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17" in 1.03802744s
9s                    Normal    Created     Pod/mongodb-0   Created container mongodb-agent-readinessprobe
9s                    Normal    Started     Pod/mongodb-0   Started container mongodb-agent-readinessprobe
7s (x2 over 9s)       Normal    Pulled      Pod/mongodb-0   Container image "docker.io/mongo:6.0.5" already present on machine
7s (x2 over 9s)       Normal    Created     Pod/mongodb-0   Created container mongod
8s                    Warning   Failed      Pod/mongodb-0   Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/rancher/rke2/agent/containerd/io.containerd.grpc.v1.cri/containers/mongod/volumes/e3e640bf3d7522afe07a18a6f13316b4629278975832a3d0edac9f2f05508e10" to rootfs at "/data/configdb": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/mongod/rootfs/data/configdb: permission denied: unknown
6s (x2 over 8s)       Normal    Pulling     Pod/mongodb-0   Pulling image "quay.io/mongodb/mongodb-agent:107.0.0.8465-1"
7s                    Normal    Pulled      Pod/mongodb-0   Successfully pulled image "quay.io/mongodb/mongodb-agent:107.0.0.8465-1" in 927.409827ms
5s (x2 over 7s)       Normal    Created     Pod/mongodb-0   Created container mongodb-agent
5s (x2 over 7s)       Warning   Failed      Pod/mongodb-0   Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/b4394964-9473-4165-96b0-12a287cf940c/volumes/kubernetes.io~empty-dir/healthstatus" to rootfs at "/var/log/mongodb-mms-automation/healthstatus": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/mongodb-agent/rootfs/var/log/mongodb-mms-automation/healthstatus: permission denied: unknown
6s                    Warning   Failed      Pod/mongodb-0   Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/rancher/rke2/agent/containerd/io.containerd.grpc.v1.cri/containers/mongod/volumes/9cf27e925b3fe7826494b5f705fa047f47f041e8ebf814e58151f413f1347b5b" to rootfs at "/data/db": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/mongod/rootfs/data/db: permission denied: unknown
5s                    Normal    Pulled      Pod/mongodb-0   Successfully pulled image "quay.io/mongodb/mongodb-agent:107.0.0.8465-1" in 899.646353ms
4s (x2 over 5s)       Warning   BackOff     Pod/mongodb-0   Back-off restarting failed container
5s                    Warning   BackOff     Pod/mongodb-0   Back-off restarting failed container
14s (x2 over 65m)     Normal    SuccessfulCreate   StatefulSet/mongodb   create Pod mongodb-0 in StatefulSet mongodb successful

Operator Information

 helm list
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
mongodb-operator        mongodb         1               2024-06-04 12:31:54.91627408 +0100 WEST deployed        community-operator-0.9.0        0.9.0      
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: mongodb
spec:
  version: "6.0.5"
  additionalMongodConfig:
    storage.wiredTiger.engineConfig.journalCompressor: zlib
...

Kubernetes Cluster Information

Additional context Looked in host /var/log/*.log and nothing relevant Used ctr utility and didn't found much, but also don't know that well how to explore it Asked for help in https://github.com/rancher/rke2/discussions/6102; Tried remove node from cluster and add it again, didn't work Forced installation in other nodes, didn't work Already installed operator with helm, and downloaded git repo installing it from there; didn't work Removed all crds, roles, etc; didn't work

Deleted and installed again, operator logs:

2024-06-04T21:28:02.405Z    INFO    controllers/replica_set_controller.go:130   Reconciling MongoDB {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.405Z    DEBUG   controllers/replica_set_controller.go:132   Validating MongoDB.Spec {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.406Z    DEBUG   controllers/replica_set_controller.go:142   Ensuring the service exists {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.406Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.419Z    INFO    controllers/replica_set_controller.go:468   Create/Update operation succeeded   {"ReplicaSet": "mongodb/mongodb", "operation": "created"}
2024-06-04T21:28:02.419Z    INFO    controllers/replica_set_controller.go:360   Creating/Updating AutomationConfig  {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.420Z    DEBUG   scram/scram.go:128  No existing credentials found, generating new credentials
2024-06-04T21:28:02.420Z    DEBUG   scram/scram.go:106  Generating new credentials and storing in secret/ops-mongodb-scram-credentials
2024-06-04T21:28:02.466Z    DEBUG   scram/scram.go:117  Successfully generated SCRAM credentials
2024-06-04T21:28:02.466Z    DEBUG   scram/scram.go:128  No existing credentials found, generating new credentials
2024-06-04T21:28:02.466Z    DEBUG   scram/scram.go:106  Generating new credentials and storing in secret/nenufares-mongodb-scram-credentials
2024-06-04T21:28:02.494Z    DEBUG   scram/scram.go:117  Successfully generated SCRAM credentials
2024-06-04T21:28:02.507Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.507Z    DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongodb-0:27017 mongodb-1:27017 mongodb-2:27017]   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.508Z    DEBUG   controllers/replica_set_controller.go:535   AutomationConfigMembersThisReconciliation   {"mdb.AutomationConfigMembersThisReconciliation()": 3}
2024-06-04T21:28:02.514Z    DEBUG   controllers/replica_set_controller.go:379   The existing StatefulSet did not have the readiness probe init container, skipping pod annotation check.    {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.514Z    INFO    controllers/replica_set_controller.go:335   Creating/Updating StatefulSet   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.524Z    INFO    controllers/replica_set_controller.go:340   Creating/Updating StatefulSet for Arbiters  {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.535Z    DEBUG   controllers/replica_set_controller.go:350   Ensuring StatefulSet is ready, with type: RollingUpdate {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.535Z    INFO    controllers/mongodb_status_options.go:110   ReplicaSet is not yet ready, retrying in 10 seconds
2024-06-04T21:28:02.547Z    INFO    controllers/replica_set_controller.go:130   Reconciling MongoDB {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.547Z    DEBUG   controllers/replica_set_controller.go:132   Validating MongoDB.Spec {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.547Z    DEBUG   controllers/replica_set_controller.go:142   Ensuring the service exists {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.548Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.554Z    INFO    controllers/replica_set_controller.go:468   Create/Update operation succeeded   {"ReplicaSet": "mongodb/mongodb", "operation": "updated"}
2024-06-04T21:28:02.555Z    INFO    controllers/replica_set_controller.go:360   Creating/Updating AutomationConfig  {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.586Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/ops-mongodb-scram-credentials
2024-06-04T21:28:02.607Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/nenufares-mongodb-scram-credentials
2024-06-04T21:28:02.607Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.607Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.607Z    DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongodb-0:27017 mongodb-1:27017 mongodb-2:27017]   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.607Z    DEBUG   controllers/replica_set_controller.go:535   AutomationConfigMembersThisReconciliation   {"mdb.AutomationConfigMembersThisReconciliation()": 3}
2024-06-04T21:28:02.608Z    DEBUG   controllers/replica_set_controller.go:383   Waiting for agents to reach version 1   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.608Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.608Z    INFO    controllers/mongodb_status_options.go:110   ReplicaSet is not yet ready, retrying in 10 seconds
2024-06-04T21:28:02.619Z    INFO    controllers/replica_set_controller.go:130   Reconciling MongoDB {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.619Z    DEBUG   controllers/replica_set_controller.go:132   Validating MongoDB.Spec {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.620Z    DEBUG   controllers/replica_set_controller.go:142   Ensuring the service exists {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.620Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.620Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.626Z    INFO    controllers/replica_set_controller.go:468   Create/Update operation succeeded   {"ReplicaSet": "mongodb/mongodb", "operation": "updated"}
2024-06-04T21:28:02.626Z    INFO    controllers/replica_set_controller.go:360   Creating/Updating AutomationConfig  {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.650Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/ops-mongodb-scram-credentials
2024-06-04T21:28:02.669Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/nenufares-mongodb-scram-credentials
2024-06-04T21:28:02.669Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.669Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.669Z    DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongodb-0:27017 mongodb-1:27017 mongodb-2:27017]   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.669Z    DEBUG   controllers/replica_set_controller.go:535   AutomationConfigMembersThisReconciliation   {"mdb.AutomationConfigMembersThisReconciliation()": 3}
2024-06-04T21:28:02.670Z    DEBUG   controllers/replica_set_controller.go:383   Waiting for agents to reach version 1   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.670Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:02.670Z    INFO    controllers/mongodb_status_options.go:110   ReplicaSet is not yet ready, retrying in 10 seconds
2024-06-04T21:28:12.548Z    INFO    controllers/replica_set_controller.go:130   Reconciling MongoDB {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.548Z    DEBUG   controllers/replica_set_controller.go:132   Validating MongoDB.Spec {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.548Z    DEBUG   controllers/replica_set_controller.go:142   Ensuring the service exists {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.548Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.548Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.557Z    INFO    controllers/replica_set_controller.go:468   Create/Update operation succeeded   {"ReplicaSet": "mongodb/mongodb", "operation": "updated"}
2024-06-04T21:28:12.558Z    INFO    controllers/replica_set_controller.go:360   Creating/Updating AutomationConfig  {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.596Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/ops-mongodb-scram-credentials
2024-06-04T21:28:12.633Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/nenufares-mongodb-scram-credentials
2024-06-04T21:28:12.634Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.634Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.634Z    DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongodb-0:27017 mongodb-1:27017 mongodb-2:27017]   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.634Z    DEBUG   controllers/replica_set_controller.go:535   AutomationConfigMembersThisReconciliation   {"mdb.AutomationConfigMembersThisReconciliation()": 3}
2024-06-04T21:28:12.635Z    DEBUG   controllers/replica_set_controller.go:383   Waiting for agents to reach version 1   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.635Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:12.635Z    INFO    controllers/mongodb_status_options.go:110   ReplicaSet is not yet ready, retrying in 10 seconds
2024-06-04T21:28:22.644Z    INFO    controllers/replica_set_controller.go:130   Reconciling MongoDB {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.644Z    DEBUG   controllers/replica_set_controller.go:132   Validating MongoDB.Spec {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.644Z    DEBUG   controllers/replica_set_controller.go:142   Ensuring the service exists {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.645Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.645Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.654Z    INFO    controllers/replica_set_controller.go:468   Create/Update operation succeeded   {"ReplicaSet": "mongodb/mongodb", "operation": "updated"}
2024-06-04T21:28:22.654Z    INFO    controllers/replica_set_controller.go:360   Creating/Updating AutomationConfig  {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.673Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/ops-mongodb-scram-credentials
2024-06-04T21:28:22.693Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/nenufares-mongodb-scram-credentials
2024-06-04T21:28:22.693Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.693Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.693Z    DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongodb-0:27017 mongodb-1:27017 mongodb-2:27017]   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.693Z    DEBUG   controllers/replica_set_controller.go:535   AutomationConfigMembersThisReconciliation   {"mdb.AutomationConfigMembersThisReconciliation()": 3}
2024-06-04T21:28:22.693Z    DEBUG   controllers/replica_set_controller.go:383   Waiting for agents to reach version 1   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.694Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:22.694Z    INFO    controllers/mongodb_status_options.go:110   ReplicaSet is not yet ready, retrying in 10 seconds
2024-06-04T21:28:32.704Z    INFO    controllers/replica_set_controller.go:130   Reconciling MongoDB {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.704Z    DEBUG   controllers/replica_set_controller.go:132   Validating MongoDB.Spec {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.704Z    DEBUG   controllers/replica_set_controller.go:142   Ensuring the service exists {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.705Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.705Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.719Z    INFO    controllers/replica_set_controller.go:468   Create/Update operation succeeded   {"ReplicaSet": "mongodb/mongodb", "operation": "updated"}
2024-06-04T21:28:32.719Z    INFO    controllers/replica_set_controller.go:360   Creating/Updating AutomationConfig  {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.740Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/ops-mongodb-scram-credentials
2024-06-04T21:28:32.759Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/nenufares-mongodb-scram-credentials
2024-06-04T21:28:32.760Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.760Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.760Z    DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongodb-0:27017 mongodb-1:27017 mongodb-2:27017]   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.760Z    DEBUG   controllers/replica_set_controller.go:535   AutomationConfigMembersThisReconciliation   {"mdb.AutomationConfigMembersThisReconciliation()": 3}
2024-06-04T21:28:32.760Z    DEBUG   controllers/replica_set_controller.go:383   Waiting for agents to reach version 1   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.760Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:32.760Z    INFO    controllers/mongodb_status_options.go:110   ReplicaSet is not yet ready, retrying in 10 seconds
2024-06-04T21:28:42.770Z    INFO    controllers/replica_set_controller.go:130   Reconciling MongoDB {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.770Z    DEBUG   controllers/replica_set_controller.go:132   Validating MongoDB.Spec {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.770Z    DEBUG   controllers/replica_set_controller.go:142   Ensuring the service exists {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.770Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.770Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.780Z    INFO    controllers/replica_set_controller.go:468   Create/Update operation succeeded   {"ReplicaSet": "mongodb/mongodb", "operation": "updated"}
2024-06-04T21:28:42.780Z    INFO    controllers/replica_set_controller.go:360   Creating/Updating AutomationConfig  {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.801Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/ops-mongodb-scram-credentials
2024-06-04T21:28:42.822Z    DEBUG   scram/scram.go:101  Credentials have not changed, using credentials stored in: secret/nenufares-mongodb-scram-credentials
2024-06-04T21:28:42.823Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.823Z    DEBUG   agent/replica_set_port_manager.go:122   No port change required {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.823Z    DEBUG   agent/replica_set_port_manager.go:40    Calculated process port map: map[mongodb-0:27017 mongodb-1:27017 mongodb-2:27017]   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.823Z    DEBUG   controllers/replica_set_controller.go:535   AutomationConfigMembersThisReconciliation   {"mdb.AutomationConfigMembersThisReconciliation()": 3}
2024-06-04T21:28:42.823Z    DEBUG   controllers/replica_set_controller.go:383   Waiting for agents to reach version 1   {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.823Z    DEBUG   agent/agent_readiness.go:106    The Pod 'mongodb-0' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "mongodb/mongodb"}
2024-06-04T21:28:42.823Z    INFO    controllers/mongodb_status_options.go:110   ReplicaSet is not yet ready, retrying in 10 seconds

And it keep going like this

k get pods       
NAME                                           READY   STATUS             RESTARTS      AGE
mongodb-0                                      0/2     CrashLoopBackOff   8 (16s ago)   119s
mongodb-kubernetes-operator-6bb747859d-6bbts   1/1     Running            0             9h

➜  k get mdbc
NAME      PHASE     VERSION
mongodb   Pending   

➜  k describe mdbc mongodb 
Name:         mongodb
Namespace:    mongodb
Labels:       <none>
Annotations:  <none>
API Version:  mongodbcommunity.mongodb.com/v1
Kind:         MongoDBCommunity
Metadata:
  Creation Timestamp:  2024-06-04T21:28:02Z
  Generation:          1
  Resource Version:    10589313
  UID:                 091360fe-d478-44a1-b5b5-6fd15c85e18b
Spec:
  Additional Mongod Config:
    storage.wiredTiger.engineConfig.journalCompressor:  zlib
  Members:                                              3
  Security:
    Authentication:
      Ignore Unknown Users:  true
      Modes:
        SCRAM
  Type:  ReplicaSet
  Users:
...
➜  kubectl get mdbc -oyaml
apiVersion: v1
items:
- apiVersion: mongodbcommunity.mongodb.com/v1
  kind: MongoDBCommunity
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"mongodbcommunity.mongodb.com/v1","kind":"MongoDBCommunity","metadata":{"annotations":{},"name":"mongodb","namespace":"mongodb"},"spec":{"additionalMongodConfig":{"storage.wiredTiger.engineConfig.journalCompressor":"zlib"},"members":3,"security":{"authentication":{"modes":["SCRAM"]}},"type":"ReplicaSet","users":[{"db":"admin","name":"boss","passwordSecretRef":{"name":"boss-mongodb-password"},"roles":[{"db":"admin","name":"clusterAdmin"},{"db":"admin","name":"userAdminAnyDatabase"}],"scramCredentialsSecretName":"boss-mongodb"},{"db":"project","name":"project","passwordSecretRef":{"name":"project-mongodb-password"},"roles":[{"db":"project","name":"dbAdmin"},{"db":"project","name":"readWrite"}],"scramCredentialsSecretName":"project-mongodb"}],"version":"6.0.5"}}
    creationTimestamp: "2024-06-04T21:28:02Z"
    generation: 1
    name: mongodb
    namespace: mongodb
    resourceVersion: "10589313"
    uid: 091360fe-d478-44a1-b5b5-6fd15c85e18b
  spec:
    additionalMongodConfig:
      storage.wiredTiger.engineConfig.journalCompressor: zlib
    members: 3
    security:
      authentication:
        ignoreUnknownUsers: true
        modes:
        - SCRAM
    type: ReplicaSet
    users:
    - db: admin
      name: boss
      passwordSecretRef:
        name: boss-mongodb-password
      roles:
      - db: admin
        name: clusterAdmin
      - db: admin
        name: userAdminAnyDatabase
      scramCredentialsSecretName: boss-mongodb
    - db: project
      name: project
      passwordSecretRef:
        name: project-mongodb-password
      roles:
      - db: project
        name: dbAdmin
      - db: project
        name: readWrite
      scramCredentialsSecretName: project-mongodb
    version: 6.0.5
  status:
    currentMongoDBMembers: 0
    currentStatefulSetReplicas: 0
    message: ReplicaSet is not yet ready, retrying in 10 seconds
    mongoUri: ""
    phase: Pending
kind: List
metadata:
  resourceVersion: ""
➜  kubectl get sts -oyaml
apiVersion: v1
items:
- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    creationTimestamp: "2024-06-04T21:28:02Z"
    generation: 1
    name: mongodb
    namespace: mongodb
    ownerReferences:
    - apiVersion: mongodbcommunity.mongodb.com/v1
      blockOwnerDeletion: true
      controller: true
      kind: MongoDBCommunity
      name: mongodb
      uid: 091360fe-d478-44a1-b5b5-6fd15c85e18b
    resourceVersion: "10589317"
    uid: 18bfc4ed-ef96-4216-9531-6cf96daa5edc
  spec:
    podManagementPolicy: OrderedReady
    replicas: 3
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: mongodb-svc
    serviceName: mongodb-svc
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: mongodb-svc
      spec:
        containers:
        - args:
          - ""
          command:
          - /bin/sh
          - -c
          - |2+

            #run post-start hook to handle version changes
            /hooks/version-upgrade

            # wait for config and keyfile to be created by the agent
             while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

            # start mongod with this configuration
            exec mongod -f /data/automation-mongod.conf;

          env:
          - name: AGENT_STATUS_FILEPATH
            value: /healthstatus/agent-health-status.json
          image: docker.io/mongo:6.0.5
          imagePullPolicy: IfNotPresent
          name: mongod
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /data
            name: data-volume
          - mountPath: /healthstatus
            name: healthstatus
          - mountPath: /hooks
            name: hooks
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: mongodb-keyfile
          - mountPath: /tmp
            name: tmp
        - command:
          - /bin/bash
          - -c
          - |-
            current_uid=$(id -u)
            AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
            declare -r current_uid
            if ! grep -q "${current_uid}" /etc/passwd ; then
            sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
            echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
            export NSS_WRAPPER_PASSWD=/tmp/passwd
            export LD_PRELOAD=libnss_wrapper.so
            export NSS_WRAPPER_GROUP=/etc/group
            fi
            agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile ${AGENT_LOG_FILE} -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS} -logLevel ${AGENT_LOG_LEVEL}
          env:
          - name: AGENT_LOG_FILE
            value: /var/log/mongodb-mms-automation/automation-agent.log
          - name: AGENT_LOG_LEVEL
            value: INFO
          - name: AGENT_MAX_LOG_FILE_DURATION_HOURS
            value: "24"
          - name: AGENT_STATUS_FILEPATH
            value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
          - name: AUTOMATION_CONFIG_MAP
            value: mongodb-config
          - name: HEADLESS_AGENT
            value: "true"
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          image: quay.io/mongodb/mongodb-agent:107.0.0.8465-1
          imagePullPolicy: Always
          name: mongodb-agent
          readinessProbe:
            exec:
              command:
              - /opt/scripts/readinessprobe
            failureThreshold: 40
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
          - mountPath: /var/lib/automation/config
            name: automation-config
            readOnly: true
          - mountPath: /data
            name: data-volume
          - mountPath: /var/log/mongodb-mms-automation/healthstatus
            name: healthstatus
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: mongodb-keyfile
          - mountPath: /tmp
            name: tmp
        dnsPolicy: ClusterFirst
        initContainers:
        - command:
          - cp
          - version-upgrade-hook
          - /hooks/version-upgrade
          image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
          imagePullPolicy: Always
          name: mongod-posthook
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /hooks
            name: hooks
        - command:
          - cp
          - /probes/readinessprobe
          - /opt/scripts/readinessprobe
          image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
          imagePullPolicy: Always
          name: mongodb-agent-readinessprobe
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 2000
          runAsNonRoot: true
          runAsUser: 2000
        serviceAccount: mongodb-database
        serviceAccountName: mongodb-database
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: agent-scripts
        - name: automation-config
          secret:
            defaultMode: 416
            secretName: mongodb-config
        - emptyDir: {}
          name: healthstatus
        - emptyDir: {}
          name: hooks
        - emptyDir: {}
          name: mongodb-keyfile
        - emptyDir: {}
          name: tmp
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10G
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 2G
        volumeMode: Filesystem
      status:
        phase: Pending
  status:
    availableReplicas: 0
    collisionCount: 0
    currentReplicas: 1
    currentRevision: mongodb-775c5cf5c7
    observedGeneration: 1
    replicas: 1
    updateRevision: mongodb-775c5cf5c7
    updatedReplicas: 1
- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    creationTimestamp: "2024-06-04T21:28:02Z"
    generation: 1
    name: mongodb-arb
    namespace: mongodb
    ownerReferences:
    - apiVersion: mongodbcommunity.mongodb.com/v1
      blockOwnerDeletion: true
      controller: true
      kind: MongoDBCommunity
      name: mongodb
      uid: 091360fe-d478-44a1-b5b5-6fd15c85e18b
    resourceVersion: "10589315"
    uid: 2b6a03ab-1412-4242-b674-0513a13c1c7a
  spec:
    podManagementPolicy: OrderedReady
    replicas: 0
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: mongodb-svc
    serviceName: mongodb-svc
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: mongodb-svc
      spec:
        containers:
        - args:
          - ""
          command:
          - /bin/sh
          - -c
          - |2+

            #run post-start hook to handle version changes
            /hooks/version-upgrade

            # wait for config and keyfile to be created by the agent
             while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

            # start mongod with this configuration
            exec mongod -f /data/automation-mongod.conf;

          env:
          - name: AGENT_STATUS_FILEPATH
            value: /healthstatus/agent-health-status.json
          image: docker.io/mongo:6.0.5
          imagePullPolicy: IfNotPresent
          name: mongod
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /data
            name: data-volume
          - mountPath: /healthstatus
            name: healthstatus
          - mountPath: /hooks
            name: hooks
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: mongodb-keyfile
          - mountPath: /tmp
            name: tmp
        - command:
          - /bin/bash
          - -c
          - |-
            current_uid=$(id -u)
            AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
            declare -r current_uid
            if ! grep -q "${current_uid}" /etc/passwd ; then
            sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
            echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
            export NSS_WRAPPER_PASSWD=/tmp/passwd
            export LD_PRELOAD=libnss_wrapper.so
            export NSS_WRAPPER_GROUP=/etc/group
            fi
            agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile ${AGENT_LOG_FILE} -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS} -logLevel ${AGENT_LOG_LEVEL}
          env:
          - name: AGENT_LOG_FILE
            value: /var/log/mongodb-mms-automation/automation-agent.log
          - name: AGENT_LOG_LEVEL
            value: INFO
          - name: AGENT_MAX_LOG_FILE_DURATION_HOURS
            value: "24"
          - name: AGENT_STATUS_FILEPATH
            value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
          - name: AUTOMATION_CONFIG_MAP
            value: mongodb-config
          - name: HEADLESS_AGENT
            value: "true"
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          image: quay.io/mongodb/mongodb-agent:107.0.0.8465-1
          imagePullPolicy: Always
          name: mongodb-agent
          readinessProbe:
            exec:
              command:
              - /opt/scripts/readinessprobe
            failureThreshold: 40
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
          - mountPath: /var/lib/automation/config
            name: automation-config
            readOnly: true
          - mountPath: /data
            name: data-volume
          - mountPath: /var/log/mongodb-mms-automation/healthstatus
            name: healthstatus
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: mongodb-keyfile
          - mountPath: /tmp
            name: tmp
        dnsPolicy: ClusterFirst
        initContainers:
        - command:
          - cp
          - version-upgrade-hook
          - /hooks/version-upgrade
          image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
          imagePullPolicy: Always
          name: mongod-posthook
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /hooks
            name: hooks
        - command:
          - cp
          - /probes/readinessprobe
          - /opt/scripts/readinessprobe
          image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
          imagePullPolicy: Always
          name: mongodb-agent-readinessprobe
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 2000
          runAsNonRoot: true
          runAsUser: 2000
        serviceAccount: mongodb-database
        serviceAccountName: mongodb-database
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: agent-scripts
        - name: automation-config
          secret:
            defaultMode: 416
            secretName: mongodb-config
        - emptyDir: {}
          name: healthstatus
        - emptyDir: {}
          name: hooks
        - emptyDir: {}
          name: mongodb-keyfile
        - emptyDir: {}
          name: tmp
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10G
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 2G
        volumeMode: Filesystem
      status:
        phase: Pending
  status:
    availableReplicas: 0
    collisionCount: 0
    currentRevision: mongodb-arb-775c5cf5c7
    observedGeneration: 1
    replicas: 0
    updateRevision: mongodb-arb-775c5cf5c7
kind: List
metadata:
  resourceVersion: ""
➜  kubectl get pods -oyaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      cni.projectcalico.org/containerID: 3a2e8d031dea7867ab3215d0bbad54472a0317521c0e9b7fcfd224ab49c5e0d9
      cni.projectcalico.org/podIP: 10.42.1.93/32
      cni.projectcalico.org/podIPs: 10.42.1.93/32
      kubernetes.io/psp: global-unrestricted-psp
    creationTimestamp: "2024-06-04T21:28:02Z"
    generateName: mongodb-
    labels:
      app: mongodb-svc
      controller-revision-hash: mongodb-775c5cf5c7
      statefulset.kubernetes.io/pod-name: mongodb-0
    name: mongodb-0
    namespace: mongodb
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: StatefulSet
      name: mongodb
      uid: 18bfc4ed-ef96-4216-9531-6cf96daa5edc
    resourceVersion: "10591586"
    uid: 8afb09cd-5286-4e0c-875c-7d74f3977d20
  spec:
    containers:
    - args:
      - ""
      command:
      - /bin/sh
      - -c
      - |2+

        #run post-start hook to handle version changes
        /hooks/version-upgrade

        # wait for config and keyfile to be created by the agent
         while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

        # start mongod with this configuration
        exec mongod -f /data/automation-mongod.conf;

      env:
      - name: AGENT_STATUS_FILEPATH
        value: /healthstatus/agent-health-status.json
      image: docker.io/mongo:6.0.5
      imagePullPolicy: IfNotPresent
      name: mongod
      resources:
        limits:
          cpu: "1"
          memory: 500M
        requests:
          cpu: 500m
          memory: 400M
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /data
        name: data-volume
      - mountPath: /healthstatus
        name: healthstatus
      - mountPath: /hooks
        name: hooks
      - mountPath: /var/log/mongodb-mms-automation
        name: logs-volume
      - mountPath: /var/lib/mongodb-mms-automation/authentication
        name: mongodb-keyfile
      - mountPath: /tmp
        name: tmp
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-fmgf9
        readOnly: true
    - command:
      - /bin/bash
      - -c
      - |-
        current_uid=$(id -u)
        AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
        declare -r current_uid
        if ! grep -q "${current_uid}" /etc/passwd ; then
        sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
        echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
        export NSS_WRAPPER_PASSWD=/tmp/passwd
        export LD_PRELOAD=libnss_wrapper.so
        export NSS_WRAPPER_GROUP=/etc/group
        fi
        agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile ${AGENT_LOG_FILE} -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS} -logLevel ${AGENT_LOG_LEVEL}
      env:
      - name: AGENT_LOG_FILE
        value: /var/log/mongodb-mms-automation/automation-agent.log
      - name: AGENT_LOG_LEVEL
        value: INFO
      - name: AGENT_MAX_LOG_FILE_DURATION_HOURS
        value: "24"
      - name: AGENT_STATUS_FILEPATH
        value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
      - name: AUTOMATION_CONFIG_MAP
        value: mongodb-config
      - name: HEADLESS_AGENT
        value: "true"
      - name: POD_NAMESPACE
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
      image: quay.io/mongodb/mongodb-agent:107.0.0.8465-1
      imagePullPolicy: Always
      name: mongodb-agent
      readinessProbe:
        exec:
          command:
          - /opt/scripts/readinessprobe
        failureThreshold: 40
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      resources:
        limits:
          cpu: "1"
          memory: 500M
        requests:
          cpu: 500m
          memory: 400M
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /opt/scripts
        name: agent-scripts
      - mountPath: /var/lib/automation/config
        name: automation-config
        readOnly: true
      - mountPath: /data
        name: data-volume
      - mountPath: /var/log/mongodb-mms-automation/healthstatus
        name: healthstatus
      - mountPath: /var/log/mongodb-mms-automation
        name: logs-volume
      - mountPath: /var/lib/mongodb-mms-automation/authentication
        name: mongodb-keyfile
      - mountPath: /tmp
        name: tmp
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-fmgf9
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    hostname: mongodb-0
    initContainers:
    - command:
      - cp
      - version-upgrade-hook
      - /hooks/version-upgrade
      image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
      imagePullPolicy: Always
      name: mongod-posthook
      resources: {}
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /hooks
        name: hooks
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-fmgf9
        readOnly: true
    - command:
      - cp
      - /probes/readinessprobe
      - /opt/scripts/readinessprobe
      image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
      imagePullPolicy: Always
      name: mongodb-agent-readinessprobe
      resources: {}
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /opt/scripts
        name: agent-scripts
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-fmgf9
        readOnly: true
    nodeName: vm05
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
      fsGroup: 2000
      runAsNonRoot: true
      runAsUser: 2000
    serviceAccount: mongodb-database
    serviceAccountName: mongodb-database
    subdomain: mongodb-svc
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: data-volume
      persistentVolumeClaim:
        claimName: data-volume-mongodb-0
    - name: logs-volume
      persistentVolumeClaim:
        claimName: logs-volume-mongodb-0
    - emptyDir: {}
      name: agent-scripts
    - name: automation-config
      secret:
        defaultMode: 416
        secretName: mongodb-config
    - emptyDir: {}
      name: healthstatus
    - emptyDir: {}
      name: hooks
    - emptyDir: {}
      name: mongodb-keyfile
    - emptyDir: {}
      name: tmp
    - name: kube-api-access-fmgf9
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2024-06-04T21:28:07Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2024-06-04T21:28:02Z"
      message: 'containers with unready status: [mongod mongodb-agent]'
      reason: ContainersNotReady
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2024-06-04T21:28:02Z"
      message: 'containers with unready status: [mongod mongodb-agent]'
      reason: ContainersNotReady
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2024-06-04T21:28:02Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://f43af8ef560aae992661e10c11c7c2c5ea569d7725e0e2f00241cd1e2648a6d8
      image: docker.io/library/mongo:6.0.5
      imageID: docker.io/library/mongo@sha256:928347070dc089a596f869a22a4204c0feace3eb03470a6a2de6814f11fb7309
      lastState:
        terminated:
          containerID: containerd://f43af8ef560aae992661e10c11c7c2c5ea569d7725e0e2f00241cd1e2648a6d8
          exitCode: 128
          finishedAt: "2024-06-04T21:34:10Z"
          message: 'failed to create containerd task: failed to create shim task:
            OCI runtime create failed: runc create failed: unable to start container
            process: error during container init: error mounting "/var/lib/rancher/rke2/agent/containerd/io.containerd.grpc.v1.cri/containers/f43af8ef560aae992661e10c11c7c2c5ea569d7725e0e2f00241cd1e2648a6d8/volumes/a9114e7cd73704f7447d9063d74dcff1e0e97d83759bb0467131030b90809ebd"
            to rootfs at "/data/db": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/f43af8ef560aae992661e10c11c7c2c5ea569d7725e0e2f00241cd1e2648a6d8/rootfs/data/db:
            permission denied: unknown'
          reason: StartError
          startedAt: "1970-01-01T00:00:00Z"
      name: mongod
      ready: false
      restartCount: 6
      started: false
      state:
        waiting:
          message: back-off 5m0s restarting failed container=mongod pod=mongodb-0_mongodb(8afb09cd-5286-4e0c-875c-7d74f3977d20)
          reason: CrashLoopBackOff
    - containerID: containerd://1888dd6cc8746b9a29281434b96f802e0d53d58adc14662ce9305b257912ede1
      image: quay.io/mongodb/mongodb-agent:107.0.0.8465-1
      imageID: quay.io/mongodb/mongodb-agent@sha256:a208e80f79bb7fe954d9a9a1444bb482dee2e86e5e5ae89dbf240395c4a158b3
      lastState:
        terminated:
          containerID: containerd://1888dd6cc8746b9a29281434b96f802e0d53d58adc14662ce9305b257912ede1
          exitCode: 128
          finishedAt: "2024-06-04T21:34:11Z"
          message: 'failed to create containerd task: failed to create shim task:
            OCI runtime create failed: runc create failed: unable to start container
            process: error during container init: error mounting "/var/lib/kubelet/pods/8afb09cd-5286-4e0c-875c-7d74f3977d20/volumes/kubernetes.io~empty-dir/healthstatus"
            to rootfs at "/var/log/mongodb-mms-automation/healthstatus": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/1888dd6cc8746b9a29281434b96f802e0d53d58adc14662ce9305b257912ede1/rootfs/var/log/mongodb-mms-automation/healthstatus:
            permission denied: unknown'
          reason: StartError
          startedAt: "1970-01-01T00:00:00Z"
      name: mongodb-agent
      ready: false
      restartCount: 6
      started: false
      state:
        waiting:
          message: back-off 5m0s restarting failed container=mongodb-agent pod=mongodb-0_mongodb(8afb09cd-5286-4e0c-875c-7d74f3977d20)
          reason: CrashLoopBackOff
    hostIP: 10.134.57.11
    initContainerStatuses:
    - containerID: containerd://52ae0977808a4aff6c47485a426e9b521c6b276ee2c29abca8238bafe5b658f2
      image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
      imageID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:b8a73d75853584a78ce9df853dd59f965c114296e10ff70f8c6d942127adc82a
      lastState: {}
      name: mongod-posthook
      ready: true
      restartCount: 0
      state:
        terminated:
          containerID: containerd://52ae0977808a4aff6c47485a426e9b521c6b276ee2c29abca8238bafe5b658f2
          exitCode: 0
          finishedAt: "2024-06-04T21:28:04Z"
          reason: Completed
          startedAt: "2024-06-04T21:28:04Z"
    - containerID: containerd://f44462680a5b82161b325bb54c1ff596dde7da230c52922478b3c3db0174d58a
      image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
      imageID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:81704c64bde4e61836af17f7b14bd009034a68d39c2399b1fad9eb1c010d0a22
      lastState: {}
      name: mongodb-agent-readinessprobe
      ready: true
      restartCount: 0
      state:
        terminated:
          containerID: containerd://f44462680a5b82161b325bb54c1ff596dde7da230c52922478b3c3db0174d58a
          exitCode: 0
          finishedAt: "2024-06-04T21:28:06Z"
          reason: Completed
          startedAt: "2024-06-04T21:28:06Z"
    phase: Running
    podIP: 10.42.1.93
    podIPs:
    - ip: 10.42.1.93
    qosClass: Burstable
    startTime: "2024-06-04T21:28:02Z"
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      cni.projectcalico.org/containerID: ad3370fdc15adfa37c454c64627361ab4d180fd5e20c63704cb34aa89ea56522
      cni.projectcalico.org/podIP: 10.42.7.16/32
      cni.projectcalico.org/podIPs: 10.42.7.16/32
      kubernetes.io/psp: global-unrestricted-psp
    creationTimestamp: "2024-06-04T12:03:18Z"
    generateName: mongodb-kubernetes-operator-6bb747859d-
    labels:
      name: mongodb-kubernetes-operator
      pod-template-hash: 6bb747859d
    name: mongodb-kubernetes-operator-6bb747859d-6bbts
    namespace: mongodb
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: mongodb-kubernetes-operator-6bb747859d
      uid: b2e4d861-76c9-4ad3-9b9a-73da2198afba
    resourceVersion: "10382863"
    uid: 7624c243-0669-4d1f-9835-241beedbcc5c
  spec:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: name
              operator: In
              values:
              - mongodb-kubernetes-operator
          topologyKey: kubernetes.io/hostname
    containers:
    - command:
      - /usr/local/bin/entrypoint
      env:
      - name: WATCH_NAMESPACE
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
      - name: POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: OPERATOR_NAME
        value: mongodb-kubernetes-operator
      - name: AGENT_IMAGE
        value: quay.io/mongodb/mongodb-agent:107.0.0.8465-1
      - name: VERSION_UPGRADE_HOOK_IMAGE
        value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.8
      - name: READINESS_PROBE_IMAGE
        value: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.17
      - name: MONGODB_IMAGE
        value: mongo
      - name: MONGODB_REPO_URL
        value: docker.io
      image: quay.io/mongodb/mongodb-kubernetes-operator:0.9.0
      imagePullPolicy: Always
      name: mongodb-kubernetes-operator
      resources:
        limits:
          cpu: 1100m
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 200Mi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-db6nb
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: vm11
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
      runAsNonRoot: true
      runAsUser: 2000
    serviceAccount: mongodb-kubernetes-operator
    serviceAccountName: mongodb-kubernetes-operator
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: kube-api-access-db6nb
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2024-06-04T12:03:18Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2024-06-04T12:03:23Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2024-06-04T12:03:23Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2024-06-04T12:03:18Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://e5ad2fcc0be79055b4c5eb331da5936e05a445129610708a54324bd200afb6ff
      image: quay.io/mongodb/mongodb-kubernetes-operator:0.9.0
      imageID: quay.io/mongodb/mongodb-kubernetes-operator@sha256:a715b07f6110c8759ddd9d6d84cede517f4ba0471f6666475df511f4c015473a
      lastState: {}
      name: mongodb-kubernetes-operator
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2024-06-04T12:03:23Z"
    hostIP: 10.134.57.42
    phase: Running
    podIP: 10.42.7.16
    podIPs:
    - ip: 10.42.7.16
    qosClass: Burstable
    startTime: "2024-06-04T12:03:18Z"
kind: List
metadata:
  resourceVersion: ""

No logs or exec into any container

Our first impression is that kubelet it's not being allowed to mount the emptyDir folders. Also stated by Brandond, rke2 maintainer. But I don't know where and how to look for it and fix it

github-actions[bot] commented 1 month ago

This issue is being marked stale because it has been open for 60 days with no activity. Please comment if this issue is still affecting you. If there is no change, this issue will be closed in 30 days.

github-actions[bot] commented 6 days ago

This issue was closed because it became stale and did not receive further updates. If the issue is still affecting you, please re-open it, or file a fresh Issue with updated information.