mongodb-agent permission denied when creating new Replica Set

What did you do to encounter the bug? Following the docs I've correctly installed the operator using the default namespace. After that I've proceeded to deploy and configure mongodb resources using this configuration file https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/config/samples/mongodb.com_v1_mongodbcommunity_cr.yaml.

What did you expect? The default example-mongodb-0 up and running.

What happened instead? While deploying the mongodb-agent I get the following error:

Failed to create file /data/automation-mongod.conf : <example-mongodb-0> [15:22:46.442] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied

I tried changed the security context from the manager.yaml but doesn't seem to apply. Maybe I'm doing something wrong?

As an update, I managed to apply the security context to the agent specifically. Apparently the securityContext set inside the manager operator deployment is not getting reflected on the images defined.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    email: support@mongodb.com
  labels:
    owner: mongodb
  name: mongodb-kubernetes-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: mongodb-kubernetes-operator
  strategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: mongodb-kubernetes-operator
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: name
                operator: In
                values:
                - mongodb-kubernetes-operator
            topologyKey: kubernetes.io/hostname
      serviceAccountName: mongodb-kubernetes-operator
      containers:
        - name: mongodb-kubernetes-operator
          image: quay.io/mongodb/mongodb-kubernetes-operator:0.7.0
          imagePullPolicy: Always
          command:
            - /usr/local/bin/entrypoint
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: mongodb-kubernetes-operator
            - name: AGENT_IMAGE
              value: quay.io/mongodb/mongodb-agent:11.0.5.6963-1
            - name: VERSION_UPGRADE_HOOK_IMAGE
              value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2
            - name: READINESS_PROBE_IMAGE
              value: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.4
            - name: MONGODB_IMAGE
              value: mongo
            - name: MONGODB_REPO_URL
              value: docker.io
          resources:
            limits:
              cpu: 1100m
              memory: 1Gi
            requests:
              cpu: 500m
              memory: 200Mi
          securityContext:
            readOnlyRootFilesystem: true
            runAsUser: 2000

What I did is setting inside the CRD a securityContext that could act with root permissions (even if this is not really safe but I'm in a closed environment for now).

---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: mongodb-specify-pod-resources
spec:
  members: 2
  type: ReplicaSet
  version: "4.4.0"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: my-user
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: my-user-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram
  statefulSet:
    spec:
      template:
        spec:
          # resources can be specified by applying an override
          # per container name.
          containers:
            - name: mongod
              resources:
                limits:
                  cpu: 1100m
                  memory: 1Gi
                requests:
                  cpu: 500m
                  memory: 200Mi
              securityContext:
                runAsNonRoot: false
                runAsUser: 0
                runAsGroup: 0
                fsGroup: 0
            - name: mongodb-agent
              resources:
                limits:
                  cpu: 1100m
                  memory: 1Gi
                requests:
                  cpu: 500m
                  memory: 200Mi
              securityContext:
                runAsNonRoot: false
                runAsUser: 0
                runAsGroup: 0
                fsGroup: 0

What I'm facing now is a different error, inside the mongod agent again.

[2021-10-09T11:28:27.597+0000] [.error] [src/action/dbcmd.go:replSetInitiateWithConf:117] <mongodb-specify-pod-resources-0> [11:28:27.597] Error running command:
    dbName=admin
    cmd=[{replSetInitiate map[_id:mongodb-specify-pod-resources members:[map[_id:0 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1] map[_id:1 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1]] protocolVersion:1]}]
    connParams=mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 (local=false) : <mongodb-specify-pod-resources-0> [11:28:27.597] Error executing WithClientFor() for cp=mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 (local=false) connectMode=SingleConnect : <mongodb-specify-pod-resources-0> [11:28:27.597] Error running command for runCommandWithTimeout(dbName=admin, cmd=[{replSetInitiate map[_id:mongodb-specify-pod-resources members:[map[_id:0 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1] map[_id:1 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1]] protocolVersion:1]}]) : result={} identityUsed=__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][668] : (InvalidReplicaSetConfig) The hosts mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 and mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 all map to this node in new configuration with {version: 1, term: 0} for replica set mongodb-specify-pod-resources
[2021-10-09T11:28:27.597+0000] [.error] [src/action/dbcmd.go:ReplSetInitiate:95] <mongodb-specify-pod-resources-0> [11:28:27.597] Error initiating repl set result=<nil> : <mongodb-specify-pod-resources-0> [11:28:27.597] Error executing WithClientFor() for cp=mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 (local=false) connectMode=SingleConnect : <mongodb-specify-pod-resources-0> [11:28:27.597] Error running command for runCommandWithTimeout(dbName=admin, cmd=[{replSetInitiate map[_id:mongodb-specify-pod-resources members:[map[_id:0 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1] map[_id:1 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1]] protocolVersion:1]}]) : result={} identityUsed=__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][668] : (InvalidReplicaSetConfig) The hosts mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 and mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 all map to this node in new configuration with {version: 1, term: 0} for replica set mongodb-specify-pod-resources

Which I really don't know how to solve this time. I'm just following step by step the documentation default instructions.

I tried forcing a schedule to specific nodes but It doesn't seem to be relevant to the matter. What I figured out is that creating the stateful set starting with one member only (members: 1) succeed. Then, doing a scale up of the set, for instance members: 3, goes well till the end (as soon as the are enough resources available). I don't know if this is expected or not, but if someone could give me a feedback it would be appreciated.

Hi @browser-bug , sorry for the late response. I tried to repo this locally and don't seem to be able to get the resource in the state you are describing.

What environment are you deploying the resource in?

As an update, I managed to apply the security context to the agent specifically. Apparently the securityContext set inside the manager operator deployment is not getting reflected on the images defined.

This is expected, the operator does not propagate its own security context to the StatefulSets that it creates, this needs to be done through the StatefulSet override.

What I'm facing now is a different error, inside the mongod agent again.

[2021-10-09T11:28:27.597+0000] [.error] [src/action/dbcmd.go:replSetInitiateWithConf:117] <mongodb-specify-pod-resources-0> [11:28:27.597] Error running command:
  dbName=admin
  cmd=[{replSetInitiate map[_id:mongodb-specify-pod-resources members:[map[_id:0 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1] map[_id:1 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1]] protocolVersion:1]}]
  connParams=mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 (local=false) : <mongodb-specify-pod-resources-0> [11:28:27.597] Error executing WithClientFor() for cp=mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 (local=false) connectMode=SingleConnect : <mongodb-specify-pod-resources-0> [11:28:27.597] Error running command for runCommandWithTimeout(dbName=admin, cmd=[{replSetInitiate map[_id:mongodb-specify-pod-resources members:[map[_id:0 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1] map[_id:1 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1]] protocolVersion:1]}]) : result={} identityUsed=__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][668] : (InvalidReplicaSetConfig) The hosts mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 and mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 all map to this node in new configuration with {version: 1, term: 0} for replica set mongodb-specify-pod-resources
[2021-10-09T11:28:27.597+0000] [.error] [src/action/dbcmd.go:ReplSetInitiate:95] <mongodb-specify-pod-resources-0> [11:28:27.597] Error initiating repl set result=<nil> : <mongodb-specify-pod-resources-0> [11:28:27.597] Error executing WithClientFor() for cp=mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 (local=false) connectMode=SingleConnect : <mongodb-specify-pod-resources-0> [11:28:27.597] Error running command for runCommandWithTimeout(dbName=admin, cmd=[{replSetInitiate map[_id:mongodb-specify-pod-resources members:[map[_id:0 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1] map[_id:1 arbiterOnly:false buildIndexes:true hidden:false host:mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 priority:1 slaveDelay:0 tags:map[] votes:1]] protocolVersion:1]}]) : result={} identityUsed=__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][668] : (InvalidReplicaSetConfig) The hosts mongodb-specify-pod-resources-0.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 and mongodb-specify-pod-resources-1.mongodb-specify-pod-resources-svc.mongodb.svc.cluster.local:27017 all map to this node in new configuration with {version: 1, term: 0} for replica set mongodb-specify-pod-resources

Which I really don't know how to solve this time. I'm just following step by step the documentation default instructions.

I'm having a similar issue related to the replica set initialization:

agent 0

k -n applianceshack logs --tail=10 as-mongodb-mayastor-001-0 mongodb-agent

[2021-11-26T14:56:34.696+0000] [.info] [src/director/director.go:computePlan:280] <as-mongodb-mayastor-001-0> [14:56:34.696] ... process has a plan : WaitAllRsMembersUp,RsInit,WaitFeatureCompatibilityVersionCorrect [2021-11-26T14:56:34.696+0000] [.info] [src/director/director.go:tracef:794] <as-mongodb-mayastor-001-0> [14:56:34.696] Running step: 'WaitAllRsMembersUp' of move 'WaitAllRsMembersUp' [2021-11-26T14:56:34.696+0000] [.info] [src/director/director.go:tracef:794] <as-mongodb-mayastor-001-0> [14:56:34.696] because [All the following are true: ['currentState.Up' = true] ['desiredState.ReplSetConf' != <nil> ('desiredState.ReplSetConf' = ReplSetConfig{id=as-mongodb-mayastor-001,version=0,commitmentStatus=false,configsvr=false,protocolVersion=1,forceProtocolVersion=false,writeConcernMajorityJournalDefault=,members={id:0,HostPort:as-mongodb-mayastor-001-0.as-mongodb-mayastor-001-svc.applianceshack.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:1,HostPort:as-mongodb-mayastor-001-1.as-mongodb-mayastor-001-svc.applianceshack.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:2,HostPort:as-mongodb-mayastor-001-2.as-mongodb-mayastor-001-svc.applianceshack.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},settings=map[]})] ] [2021-11-26T14:56:34.697+0000] [.info] [src/director/director.go:planAndExecute:564] <as-mongodb-mayastor-001-0> [14:56:34.697] Step=WaitAllRsMembersUp as part of Move=WaitAllRsMembersUp in plan failed : <as-mongodb-mayastor-001-0> [14:56:34.697] Postcondition not yet met for step WaitAllRsMembersUp because ['currentState.Fickle.AllRsMembersUp() ' = false]. Recomputing a plan...

agent 1

k -n applianceshack logs --tail=10 as-mongodb-mayastor-001-1 mongodb-agent

['currentState.CanRsInit' = false] ['desiredState.ReplSetConf' != <nil> ('desiredState.ReplSetConf' = ReplSetConfig{id=as-mongodb-mayastor-001,version=0,commitmentStatus=false,configsvr=false,protocolVersion=1,forceProtocolVersion=false,writeConcernMajorityJournalDefault=,members={id:0,HostPort:as-mongodb-mayastor-001-0.as-mongodb-mayastor-001-svc.applianceshack.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:1,HostPort:as-mongodb-mayastor-001-1.as-mongodb-mayastor-001-svc.applianceshack.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:2,HostPort:as-mongodb-mayastor-001-2.as-mongodb-mayastor-001-svc.applianceshack.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},settings=map[]})] ['currentState.ReplSetConf' = <nil>] ] [2021-11-26T14:56:38.694+0000] [.info] [src/director/director.go:planAndExecute:564] <as-mongodb-mayastor-001-1> [14:56:38.693] Step=WaitRsInit as part of Move=WaitRsInit in plan failed : <as-mongodb-mayastor-001-1> [14:56:38.693] Postcondition not yet met for step WaitRsInit because ['currentState.ReplSetConf' = <nil>]. Recomputing a plan... [2021-11-26T14:56:39.055+0000] [.info] [src/runtimestats/printer.go:mainLoop:58] <runtimestatsPrinter> [14:56:39.055] memory:rss=63410176 alloc=10189728 totalalloc=27888320000 mallocs=569895348 frees=569819655 heapinuse=13139968 heapobjects=75693 gcpausetotalns=958864583 gcpauselastns=117730 gcnum=7126 utimens=644053669000 stimens=99629789000 minflt=10206majflt=0 [2021-11-26T14:56:39.237+0000] [.info] [main/components/agent.go:LoadClusterConfig:262] [14:56:39.237] clusterConfig unchanged

agent 2

k -n applianceshack logs --tail=10 as-mongodb-mayastor-001-2 mongodb-agent [All the following are true: ['currentState.Up' = true] ['currentState.CanRsInit' = false] ['desiredState.ReplSetConf' != <nil> ('desiredState.ReplSetConf' = ReplSetConfig{id=as-mongodb-mayastor-001,version=0,commitmentStatus=false,configsvr=false,protocolVersion=1,forceProtocolVersion=false,writeConcernMajorityJournalDefault=,members={id:0,HostPort:as-mongodb-mayastor-001-0.as-mongodb-mayastor-001-svc.applianceshack.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:1,HostPort:as-mongodb-mayastor-001-1.as-mongodb-mayastor-001-svc.applianceshack.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:2,HostPort:as-mongodb-mayastor-001-2.as-mongodb-mayastor-001-svc.applianceshack.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},settings=map[]})] ['currentState.ReplSetConf' = <nil>] ] [2021-11-26T14:56:42.299+0000] [.info] [src/director/director.go:planAndExecute:564] <as-mongodb-mayastor-001-2> [14:56:42.299] Step=WaitRsInit as part of Move=WaitRsInit in plan failed : <as-mongodb-mayastor-001-2> [14:56:42.299] Postcondition not yet met for step WaitRsInit because ['currentState.ReplSetConf' = <nil>]. Recomputing a plan... [2021-11-26T14:56:43.150+0000] [.info] [main/components/agent.go:LoadClusterConfig:262] [14:56:43.150] clusterConfig unchanged

The target config seems to be formatted in this output without the element separator (comma). ,ArbiterOnly:falsePriority:1,

I have exactly the same issue, with mongodb-agent stating that it cannot open the logfile /var/log/mongodb-mms-automation/readiness.log The mongod container seems to remain in a waiting state, and nothing really starts. Here are other logs from mongo-agent saying that they cannot access anything on any volume, emptyDir or Persistent Volume.

[2021-12-08T14:13:26.655+0000] [.error] [src/mongoctl/processctl.go:createConfFile:745] <mongodb-0> [14:13:26.655] Failed to create file /data/automation-mongod.conf : <mongodb-0> [14:13:26.655] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied
[2021-12-08T14:13:26.655+0000] [.error] [src/mongoctl/processctl.go:newStartProcessCmd:459] <mongodb-0> [14:13:26.655] Failed to create conf file : <mongodb-0> [14:13:26.655] Failed to create file /data/automation-mongod.conf : <mongodb-0> [14:13:26.655] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied
[2021-12-08T14:13:26.655+0000] [.error] [src/mongoctl/processctl.go:StartMongoInBin:141] [103] <mongodb-0> [14:13:26.655] Error getting start process cmd for executable=mongod, stip=[args={"net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"authorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}[],confPath=/data/automation-mongod.conf,version=4.2.6-(),isKmipRotateMasterKey=false,useOldConfFile=false] : <mongodb-0> [14:13:26.655] Failed to create conf file : <mongodb-0> [14:13:26.655] Failed to create file /data/automation-mongod.conf : <mongodb-0> [14:13:26.655] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied
[2021-12-08T14:13:26.655+0000] [.error] [src/action/start.go:func1:93] [103] <mongodb-0> [14:13:26.655] Error starting mongod : <mongodb-0> [14:13:26.655] Error getting start process cmd for executable=mongod, stip=[args={"net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"authorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}[],confPath=/data/automation-mongod.conf,version=4.2.6-(),isKmipRotateMasterKey=false,useOldConfFile=false] : <mongodb-0> [14:13:26.655] Failed to create conf file : <mongodb-0> [14:13:26.655] Failed to create file /data/automation-mongod.conf : <mongodb-0> [14:13:26.655] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied
[2021-12-08T14:13:26.655+0000] [.error] [src/director/director.go:executePlan:962] <mongodb-0> [14:13:26.655] Failed to apply action. Result = <nil> : <mongodb-0> [14:13:26.655] Error starting mongod : <mongodb-0> [14:13:26.655] Error getting start process cmd for executable=mongod, stip=[args={"net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"authorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}[],confPath=/data/automation-mongod.conf,version=4.2.6-(),isKmipRotateMasterKey=false,useOldConfFile=false] : <mongodb-0> [14:13:26.655] Failed to create conf file : <mongodb-0> [14:13:26.655] Failed to create file /data/automation-mongod.conf : <mongodb-0> [14:13:26.655] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied
[2021-12-08T14:13:26.655+0000] [.error] [src/director/director.go:planAndExecute:567] <mongodb-0> [14:13:26.655] Plan execution failed on step StartFresh as part of move Start : <mongodb-0> [14:13:26.655] Failed to apply action. Result = <nil> : <mongodb-0> [14:13:26.655] Error starting mongod : <mongodb-0> [14:13:26.655] Error getting start process cmd for executable=mongod, stip=[args={"net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"authorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}[],confPath=/data/automation-mongod.conf,version=4.2.6-(),isKmipRotateMasterKey=false,useOldConfFile=false] : <mongodb-0> [14:13:26.655] Failed to create conf file : <mongodb-0> [14:13:26.655] Failed to create file /data/automation-mongod.conf : <mongodb-0> [14:13:26.655] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied
[2021-12-08T14:13:26.655+0000] [.error] [src/director/director.go:mainLoop:399] <mongodb-0> [14:13:26.655] Failed to planAndExecute : <mongodb-0> [14:13:26.655] Plan execution failed on step StartFresh as part of move Start : <mongodb-0> [14:13:26.655] Failed to apply action. Result = <nil> : <mongodb-0> [14:13:26.655] Error starting mongod : <mongodb-0> [14:13:26.655] Error getting start process cmd for executable=mongod, stip=[args={"net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"authorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}[],confPath=/data/automation-mongod.conf,version=4.2.6-(),isKmipRotateMasterKey=false,useOldConfFile=false] : <mongodb-0> [14:13:26.655] Failed to create conf file : <mongodb-0> [14:13:26.655] Failed to create file /data/automation-mongod.conf : <mongodb-0> [14:13:26.655] Error creating /data/automation-mongod.conf : open /data/automation-mongod.conf: permission denied

I tried to use the template given in the docs to create the cluster, and I just modified the password field.

When attaching a shell to the container I found out that the directories don't have the right permissions for user 2000 :

Here the file mongodb-mms-automation/readiness.log cannot be created

I have no name!@mongodb-0:/var/log/mongodb-mms-automation$ ls -liah
total 28K
  2 drwxr-xr-x 4 root root 4.0K Dec  8 14:11 .
2075021 drwxr-xr-x 1 root root 4.0K Jul 29 04:02 ..
1816872 drwxrwsrwx 2 root 2000 4.0K Dec  8 14:18 healthstatus
 11 drwx------ 2 root root  16K Dec  8 14:10 lost+found

And here the data directory and subdirectories also do have rights for root only

I have no name!@mongodb-0:/data$ ls -liah
total 32K
  2 drwxr-xr-x 5 root root 4.0K Dec  8 14:11 .
2075050 drwxr-xr-x 1 root root 4.0K Dec  8 14:11 ..
524289 drwxr-xr-x 2 root root 4.0K Dec  8 14:11 configdb
131073 drwxr-xr-x 2 root root 4.0K Dec  8 14:11 db
 11 drwx------ 2 root root  16K Dec  8 14:10 lost+found

Hi everyone,

There's a sample YAML file that will change the permissions of the /data directory when starting the Pods,

https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/config/samples/arbitrary_statefulset_configuration/mongodb.com_v1_hostpath.yaml

That example helps you set up hostPath volumes (which is not your case) but the strategy should be similar to make a directory owned by the Pod.

Now I have a few questions for you (all of you):

Where are you deploying this MongoDB resource (which Kubernetes provider are you using). Is this Kind, or minikube? A cloud provider Kubernetes offering? GKS, EKS?
I'm assuming you are using the default storageClass. Is this correct? If not, is there anything in particular with that storage class that could be of interest?

We assume the storage-class you are using will spawn a Volume for a PersistentVolumeClaim that, when mounted to the Pod, will allow for write-access. The MongoDB Pods are configured with ReadWriteOnce mode.

Please paste the output of the following command:

kubectl get/mongodb-specify-pod-resources sts -o yaml

To confirm that the Pods have been configured correctly.

The following has been working without issue since December 31.

kubectl get sts mongodb-mayastor -o yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-12-31T20:43:11Z"
  generation: 1
  labels:
    app: mongodb-mayastor-svc
  name: mongodb-mayastor
  namespace: applianceshack
  ownerReferences:
    - apiVersion: mongodbcommunity.mongodb.com/v1
      blockOwnerDeletion: true
      controller: true
      kind: MongoDBCommunity
      name: mongodb-mayastor
      uid: 629eb39a-7ca0-4a74-a243-b41fcf72af35
  resourceVersion: "16187151"
  uid: a491138f-dc86-48d6-b4a7-0b58e7679ade
spec:
  podManagementPolicy: OrderedReady
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: mongodb-mayastor-svc
  serviceName: mongodb-mayastor-svc
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: mongodb-mayastor-svc
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: openebs.io/engine
                    operator: In
                    values:
                      - mayastor
      containers:
        - command:
            - /bin/sh
            - -c
            -2+
              #run post-start hook to handle version changes
              /hooks/version-upgrade

              # wait for config and keyfile to be created by the agent
               while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

              # with mongod configured to append logs, we need to provide them to stdout as
              # mongod does not write to stdout and a log file
              tail -F /var/log/mongodb-mms-automation/mongodb.log > /dev/stdout &

              # start mongod with this configuration
              exec mongod -f /data/automation-mongod.conf;

          env:
            - name: AGENT_STATUS_FILEPATH
              value: /healthstatus/agent-health-status.json
          image: docker.io/mongo:4.2.6
          imagePullPolicy: IfNotPresent
          name: mongod
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            runAsNonRoot: true
            runAsUser: 2000
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /data
              name: data-volume
            - mountPath: /healthstatus
              name: healthstatus
            - mountPath: /hooks
              name: hooks
            - mountPath: /var/log/mongodb-mms-automation
              name: logs-volume
            - mountPath: /var/lib/mongodb-mms-automation/authentication
              name: mongodb-mayastor-keyfile
        - command:
            - /bin/bash
            - -c
            - |-
              current_uid=$(id -u)
              AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
              declare -r current_uid
              if ! grep -q "${current_uid}" /etc/passwd ; then
              sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
              echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
              export NSS_WRAPPER_PASSWD=/tmp/passwd
              export LD_PRELOAD=libnss_wrapper.so
              export NSS_WRAPPER_GROUP=/etc/group
              fi
              agent/mongodb-agent -cluster=/var/lib/automation/config/cluster-config.json -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -skipMongoStart -noDaemonize -useLocalMongoDbTools
          env:
            - name: AGENT_STATUS_FILEPATH
              value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
            - name: AUTOMATION_CONFIG_MAP
              value: mongodb-mayastor-config
            - name: HEADLESS_AGENT
              value: "true"
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
          image: quay.io/mongodb/mongodb-agent:11.0.5.6963-1
          imagePullPolicy: Always
          name: mongodb-agent
          readinessProbe:
            exec:
              command:
                - /opt/scripts/readinessprobe
            failureThreshold: 5000
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            runAsNonRoot: true
            runAsUser: 2000
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /opt/scripts
              name: agent-scripts
            - mountPath: /var/lib/automation/config
              name: automation-config
              readOnly: true
            - mountPath: /data
              name: data-volume
            - mountPath: /var/log/mongodb-mms-automation/healthstatus
              name: healthstatus
            - mountPath: /var/log/mongodb-mms-automation
              name: logs-volume
            - mountPath: /var/lib/mongodb-mms-automation/authentication
              name: mongodb-mayastor-keyfile
      dnsPolicy: ClusterFirst
      initContainers:
        - command:
            - cp
            - version-upgrade-hook
            - /hooks/version-upgrade
          image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.3
          imagePullPolicy: Always
          name: mongod-posthook
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /hooks
              name: hooks
        - command:
            - cp
            - /probes/readinessprobe
            - /opt/scripts/readinessprobe
          image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.6
          imagePullPolicy: Always
          name: mongodb-agent-readinessprobe
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /opt/scripts
              name: agent-scripts
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 2000
      serviceAccount: mongodb-database
      serviceAccountName: mongodb-database
      terminationGracePeriodSeconds: 30
      volumes:
        - emptyDir: {}
          name: agent-scripts
        - name: automation-config
          secret:
            defaultMode: 416
            secretName: mongodb-mayastor-config
        - emptyDir: {}
          name: healthstatus
        - emptyDir: {}
          name: hooks
        - emptyDir: {}
          name: mongodb-mayastor-keyfile
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: mayastor-1
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: mayastor-1
        volumeMode: Filesystem
      status:
        phase: Pending
status:
  availableReplicas: 3
  collisionCount: 0
  currentReplicas: 3
  currentRevision: mongodb-mayastor-5664fbdb9b
  observedGeneration: 1
  readyReplicas: 3
  replicas: 3
  updateRevision: mongodb-mayastor-5664fbdb9b
  updatedReplicas: 3

apiVersion: v1
kind: Pod
metadata:
  annotations:
    agent.mongodb.com/version: "1"
    kubernetes.io/psp: privileged
  creationTimestamp: "2021-12-31T20:43:11Z"
  generateName: mongodb-mayastor-
  labels:
    app: mongodb-mayastor-svc
    controller-revision-hash: mongodb-mayastor-5664fbdb9b
    statefulset.kubernetes.io/pod-name: mongodb-mayastor-0
  name: mongodb-mayastor-0
  namespace: applianceshack
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: mongodb-mayastor
    uid: a491138f-dc86-48d6-b4a7-0b58e7679ade
  resourceVersion: "16187064"
  uid: f795d1dd-4ff4-4045-a179-0feeabe0f389
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: openebs.io/engine
            operator: In
            values:
            - mayastor
  containers:
  - command:
    - /bin/sh
    - -c
    - |2+

      #run post-start hook to handle version changes
      /hooks/version-upgrade

      # wait for config and keyfile to be created by the agent
       while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

      # with mongod configured to append logs, we need to provide them to stdout as
      # mongod does not write to stdout and a log file
      tail -F /var/log/mongodb-mms-automation/mongodb.log > /dev/stdout &

      # start mongod with this configuration
      exec mongod -f /data/automation-mongod.conf;

    env:
    - name: AGENT_STATUS_FILEPATH
      value: /healthstatus/agent-health-status.json
    image: docker.io/mongo:4.2.6
    imagePullPolicy: IfNotPresent
    name: mongod
    resources:
      limits:
        cpu: "1"
        memory: 500M
      requests:
        cpu: 500m
        memory: 400M
    securityContext:
      runAsNonRoot: true
      runAsUser: 2000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /data
      name: data-volume
    - mountPath: /healthstatus
      name: healthstatus
    - mountPath: /hooks
      name: hooks
    - mountPath: /var/log/mongodb-mms-automation
      name: logs-volume
    - mountPath: /var/lib/mongodb-mms-automation/authentication
      name: mongodb-mayastor-keyfile
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-866cm
      readOnly: true
  - command:
    - /bin/bash
    - -c
    - |-
      current_uid=$(id -u)
      AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
      declare -r current_uid
      if ! grep -q "${current_uid}" /etc/passwd ; then
      sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
      echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
      export NSS_WRAPPER_PASSWD=/tmp/passwd
      export LD_PRELOAD=libnss_wrapper.so
      export NSS_WRAPPER_GROUP=/etc/group
      fi
      agent/mongodb-agent -cluster=/var/lib/automation/config/cluster-config.json -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -skipMongoStart -noDaemonize -useLocalMongoDbTools
    env:
    - name: AGENT_STATUS_FILEPATH
      value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
    - name: AUTOMATION_CONFIG_MAP
      value: mongodb-mayastor-config
    - name: HEADLESS_AGENT
      value: "true"
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    image: quay.io/mongodb/mongodb-agent:11.0.5.6963-1
    imagePullPolicy: Always
    name: mongodb-agent
    readinessProbe:
      exec:
        command:
        - /opt/scripts/readinessprobe
      failureThreshold: 5000
      initialDelaySeconds: 30
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "1"
        memory: 500M
      requests:
        cpu: 500m
        memory: 400M
    securityContext:
      runAsNonRoot: true
      runAsUser: 2000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/scripts
      name: agent-scripts
    - mountPath: /var/lib/automation/config
      name: automation-config
      readOnly: true
    - mountPath: /data
      name: data-volume
    - mountPath: /var/log/mongodb-mms-automation/healthstatus
      name: healthstatus
    - mountPath: /var/log/mongodb-mms-automation
      name: logs-volume
    - mountPath: /var/lib/mongodb-mms-automation/authentication
      name: mongodb-mayastor-keyfile
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-866cm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: mongodb-mayastor-0
  initContainers:
  - command:
    - cp
    - version-upgrade-hook
    - /hooks/version-upgrade
    image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.3
    imagePullPolicy: Always
    name: mongod-posthook
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /hooks
      name: hooks
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-866cm
      readOnly: true
  - command:
    - cp
    - /probes/readinessprobe
    - /opt/scripts/readinessprobe
    image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.6
    imagePullPolicy: Always
    name: mongodb-agent-readinessprobe
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/scripts
      name: agent-scripts
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-866cm
      readOnly: true
  nodeName: bl-work-mayastor-a2
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 2000
  serviceAccount: mongodb-database
  serviceAccountName: mongodb-database
  subdomain: mongodb-mayastor-svc
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: data-volume
    persistentVolumeClaim:
      claimName: data-volume-mongodb-mayastor-0
  - name: logs-volume
    persistentVolumeClaim:
      claimName: logs-volume-mongodb-mayastor-0
  - emptyDir: {}
    name: agent-scripts
  - name: automation-config
    secret:
      defaultMode: 416
      secretName: mongodb-mayastor-config
  - emptyDir: {}
    name: healthstatus
  - emptyDir: {}
    name: hooks
  - emptyDir: {}
    name: mongodb-mayastor-keyfile
  - name: kube-api-access-866cm
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-12-31T20:43:29Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-12-31T20:44:43Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2021-12-31T20:44:43Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-12-31T20:43:12Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://74dd91ee75881853ea6219988f687367a05669ca1ce0f873aa7f8394ae46d24a
    image: docker.io/library/mongo:4.2.6
    imageID: docker.io/library/mongo@sha256:c880f6b56f443bb4d01baa759883228cd84fa8d78fa1a36001d1c0a0712b5a07
    lastState: {}
    name: mongod
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-12-31T20:43:49Z"
  - containerID: containerd://fac5ab30b3b3833fb0a5c2de396c31fbab01d531e77d9782ca760864d1924862
    image: quay.io/mongodb/mongodb-agent:11.0.5.6963-1
    imageID: quay.io/mongodb/mongodb-agent@sha256:3cae7536e3557001014ebcfc328c60c9e55629e6f3788c8850767a54f43b7d95
    lastState: {}
    name: mongodb-agent
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-12-31T20:44:12Z"
  hostIP: 10.5.4.106
  initContainerStatuses:
  - containerID: containerd://fdd26dacacfb6aacdcd70bc143f874107ac0b31f22206895ceb72717f320adef
    image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.3
    imageID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:6fe1df13c21f71d30b4fac9cb9184cd06324d6c6f12a472b298754429d630ce3
    lastState: {}
    name: mongod-posthook
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://fdd26dacacfb6aacdcd70bc143f874107ac0b31f22206895ceb72717f320adef
        exitCode: 0
        finishedAt: "2021-12-31T20:43:25Z"
        reason: Completed
        startedAt: "2021-12-31T20:43:25Z"
  - containerID: containerd://09d4b9c8564a88e452859c002a4400f0bedb9902f14c668a6b75c9c00d923900
    image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.6
    imageID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:e935e27bd88133f17fdb207026e8718b9970d872483195fe57407da44280baf9
    lastState: {}
    name: mongodb-agent-readinessprobe
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://09d4b9c8564a88e452859c002a4400f0bedb9902f14c668a6b75c9c00d923900
        exitCode: 0
        finishedAt: "2021-12-31T20:43:28Z"
        reason: Completed
        startedAt: "2021-12-31T20:43:28Z"
  phase: Running
  podIP: 10.244.13.10
  podIPs:
  - ip: 10.244.13.10
  qosClass: Burstable
  startTime: "2021-12-31T20:43:12Z"

@mrik974 Were you able to resolve the issue with readiness.log cannot be created. I am stuck on the same write error: can't open new logfile: open /var/log/mongodb-mms-automation/readiness.log: permission denied

We haven't been able to reproduce this issue. If it's still affecting you, please file a fresh ticket

I encounter this issue on Rancher/RKE2, however only sporadically during operation, not during the initial creation of the container. The storage backend is Longhorn, which reports all volumes to be "Healthy".

Unhealthy
Pod mongodb-replset-2 (combined from similar events):
Readiness probe failed:
2022-07-18  09:33:57.177452702 +0000 UTC m=+0.015650687 write error: can't rename  log file: rename /var/log/mongodb-mms-automation/readiness.log  /var/log/mongodb-mms-automation/readiness-2022-07-18T09-33-57.177.log:  read-only file system 2022-07-18 09:33:57.177675385 +0000 UTC m=+0.015873346 write error:  can't rename log file: rename  /var/log/mongodb-mms-automation/readiness.log  /var/log/mongodb-mms-automation/readiness-2022-07-18T09-33-57.177.log:  read-only file system

Trying to write inside the container confirms the issue:

In mongodb-replset-2, the failing one:

$ whoami
whoami: cannot find name for user ID 2000
$ mount |head -n1
overlay on / type overlay (rw,relatime,context="system_u:object_r:container_file_t:s0:c803,c971",lowerdir=/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/275/fs:/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/274/fs:/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/272/fs:/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/271/fs,upperdir=/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/276/fs,workdir=/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/276/work)
$ echo "foo" >> /var/log/mongodb-mms-automation/readiness.log
sh: 7: cannot create /var/log/mongodb-mms-automation/readiness.log: Read-only file system

In mongodb-replset-1, one of the working ones:

$ whoami
whoami: cannot find name for user ID 2000
$ mount |head -n1
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/782/fs:/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/781/fs:/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/780/fs:/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/300/fs,upperdir=/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/783/fs,workdir=/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/783/work)
$ echo "foo" >> /var/log/mongodb-mms-automation/readiness.log
$ echo $?
0

The directory seems to be "correctly" owned by the unmapped 2000 UID:

`/var/log/` inside `mongodb-replset-2`

``` $ kubectl -n=infra-mongodb exec -it mongodb-replset-2 -- sh Defaulted container "mongodb-enterprise-database" out of: mongodb-enterprise-database, mongodb-enterprise-init-database (init) $ ls -ahl /var/log total 320K drwxr-xr-x. 1 root root 44 Jul 17 03:21 . drwxr-xr-x. 1 root root 6 May 31 11:55 .. -rw-r--r--. 1 root root 3.7K May 31 11:55 alternatives.log drwxr-xr-x. 1 root root 60 Jul 17 03:21 apt -rw-r--r--. 1 root root 35K May 31 11:54 bootstrap.log -rw-rw----. 1 root utmp 0 May 31 11:54 btmp -rw-r--r--. 1 root root 232K Jul 17 03:21 dpkg.log -rw-r--r--. 1 root root 3.2K May 31 11:54 faillog -rw-rw-r--. 1 root utmp 29K May 31 11:54 lastlog drwxrwsr-x. 4 root 2000 4.0K Jul 18 01:06 mongodb-mms-automation -rw-------. 1 root root 6.4K May 31 11:54 tallylog -rw-rw-r--. 1 root utmp 0 May 31 11:54 wtmp $ ls -ahl /var/log/mongodb-mms-automation total 33M drwxrwsr-x. 4 root 2000 4.0K Jul 18 01:06 . drwxr-xr-x. 1 root root 44 Jul 17 03:21 .. -rw-rw-r--. 1 2000 2000 50 Jul 17 21:53 .slogger-state-automation-agent-verbose.log -rw-rw-r--. 1 2000 2000 50 Jul 17 21:53 .slogger-state-automation-agent.log -rw-rw-r--. 1 2000 2000 50 Jul 17 21:53 .slogger-state-backup-agent.log -rw-rw-r--. 1 2000 2000 50 Jul 17 21:53 .slogger-state-monitoring-agent.log -rw-------. 1 2000 2000 1.5K Jul 18 01:06 agent-health-status.json -rw-rw-r--. 1 2000 2000 602 Jul 17 21:53 agent-launcher-script.log -rw-rw-r--. 1 2000 2000 952 Jul 17 21:53 automation-agent-stderr.log -rw-r--r--. 1 2000 2000 1.9M Jul 18 01:06 automation-agent-verbose.log -rw-rw-r--. 1 2000 2000 0 Jul 15 20:45 automation-agent-verbose.log.2022-07-15T20-45-26 -rw-rw-r--. 1 2000 2000 1.7K Jul 15 20:45 automation-agent-verbose.log.2022-07-15T20-45-26-1 -rw-rw-r--. 1 2000 2000 642K Jul 15 22:03 automation-agent-verbose.log.2022-07-15T22-03-49 -rw-rw-r--. 1 2000 2000 1.7K Jul 15 22:03 automation-agent-verbose.log.2022-07-15T22-03-50 -rw-rw-r--. 1 2000 2000 273K Jul 16 13:44 automation-agent-verbose.log.2022-07-16T13-44-42 -rw-rw-r--. 1 2000 2000 1.7K Jul 16 13:44 automation-agent-verbose.log.2022-07-16T13-44-42-1 -rw-rw-r--. 1 2000 2000 15M Jul 17 13:44 automation-agent-verbose.log.2022-07-17T13-44-44 -rw-rw-r--. 1 2000 2000 5.2M Jul 17 21:53 automation-agent-verbose.log.2022-07-17T21-53-35 -rw-r--r--. 1 2000 2000 1.7K Jul 17 21:53 automation-agent-verbose.log.2022-07-17T21-53-36 -rw-r--r--. 1 2000 2000 2.1K Jul 17 21:53 automation-agent.log -rw-rw-r--. 1 2000 2000 886 Jul 15 20:45 automation-agent.log.2022-07-15T20-45-26 -rw-rw-r--. 1 2000 2000 12K Jul 15 21:51 automation-agent.log.2022-07-15T22-03-49 -rw-rw-r--. 1 2000 2000 886 Jul 15 22:03 automation-agent.log.2022-07-15T22-03-50 -rw-rw-r--. 1 2000 2000 2.1K Jul 15 22:04 automation-agent.log.2022-07-16T13-44-42 -rw-rw-r--. 1 2000 2000 886 Jul 16 13:44 automation-agent.log.2022-07-16T13-44-42-1 -rw-rw-r--. 1 2000 2000 3.6K Jul 17 21:43 automation-agent.log.2022-07-17T21-43-24 -rw-rw-r--. 1 2000 2000 521K Jul 17 21:45 automation-agent.log.2022-07-17T21-53-36 -rw-r--r--. 1 2000 2000 886 Jul 17 21:53 automation-agent.log.2022-07-17T21-53-36-1 -rw-r--r--. 1 2000 2000 35K Jul 18 01:05 backup-agent.log -rw-rw-r--. 1 2000 2000 13K Jul 15 21:51 backup-agent.log.2022-07-15T22-03-51 -rw-rw-r--. 1 2000 2000 6.4K Jul 15 22:27 backup-agent.log.2022-07-16T13-44-43 -rw-rw-r--. 1 2000 2000 244K Jul 17 13:44 backup-agent.log.2022-07-17T13-44-45 -rw-rw-r--. 1 2000 2000 83K Jul 17 21:44 backup-agent.log.2022-07-17T21-53-36 drwxrwsr-x. 2 2000 2000 4.0K Jul 18 01:06 filebeat drwxrws---. 2 root 2000 16K Jul 15 20:43 lost+found -rw-------. 1 2000 2000 2.5M Jul 18 01:06 mongodb.log -rw-r--r--. 1 2000 2000 23 Jul 15 20:45 mongodb.log.2022-07-15T20-46-39.gz -rw-r--r--. 1 2000 2000 82K Jul 15 22:03 mongodb.log.2022-07-15T22-04-04.gz -rw-r--r--. 1 2000 2000 26K Jul 16 13:44 mongodb.log.2022-07-16T13-45-02.gz -rw-r--r--. 1 2000 2000 1.2M Jul 17 13:46 mongodb.log.2022-07-17T13-46-42.gz -rw-r--r--. 1 2000 2000 375K Jul 17 21:43 mongodb.log.2022-07-17T21-43-24.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-28.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-32.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-35.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-38.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-41.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-44.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-48.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-51.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-55.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:43 mongodb.log.2022-07-17T21-43-58.gz -rw-r--r--. 1 2000 2000 1.9K Jul 17 21:44 mongodb.log.2022-07-17T21-44-07.gz -rw-r--r--. 1 2000 2000 1.8K Jul 17 21:44 mongodb.log.2022-07-17T21-44-18.gz -rw-r--r--. 1 2000 2000 1.8K Jul 17 21:44 mongodb.log.2022-07-17T21-44-33.gz -rw-rw----. 1 2000 2000 5.4K Jul 17 21:44 mongodb.log.2022-07-17T21-44-49 -rw-rw----. 1 2000 2000 5.4K Jul 17 21:44 mongodb.log.2022-07-17T21-45-10 -rw-rw----. 1 2000 2000 5.4K Jul 17 21:45 mongodb.log.2022-07-17T21-45-32 -rw-rw----. 1 2000 2000 3.7K Jul 17 21:53 mongodb.log.2022-07-17T21-53-50 -rw-r--r--. 1 2000 2000 59K Jul 18 01:06 monitoring-agent.log -rw-rw-r--. 1 2000 2000 52K Jul 15 21:52 monitoring-agent.log.2022-07-15T22-03-50 -rw-rw-r--. 1 2000 2000 12K Jul 15 22:27 monitoring-agent.log.2022-07-16T13-44-43 -rw-rw-r--. 1 2000 2000 404K Jul 17 13:45 monitoring-agent.log.2022-07-17T13-45-01 -rw-rw-r--. 1 2000 2000 139K Jul 17 21:45 monitoring-agent.log.2022-07-17T21-53-36 -rw-rw-r--. 1 2000 2000 5.1M Jul 18 01:06 readiness.log ```

It seems to be the same in the other container:

`/var/log/` inside `mongodb-replset-1`

``` $ kubectl -n=infra-mongodb exec -it mongodb-replset-1 -- sh Defaulted container "mongodb-enterprise-database" out of: mongodb-enterprise-database, mongodb-enterprise-init-database (init) $ ls -ahl /var/log total 336K drwxr-xr-x 1 root root 4.0K Jul 16 03:19 . drwxr-xr-x 1 root root 4.0K May 31 11:55 .. -rw-r--r-- 1 root root 3.7K May 31 11:55 alternatives.log drwxr-xr-x 1 root root 4.0K Jul 16 03:19 apt -rw-r--r-- 1 root root 35K May 31 11:54 bootstrap.log -rw-rw---- 1 root utmp 0 May 31 11:54 btmp -rw-r--r-- 1 root root 232K Jul 16 03:19 dpkg.log -rw-r--r-- 1 root root 3.2K May 31 11:54 faillog -rw-rw-r-- 1 root utmp 29K May 31 11:54 lastlog drwxrwsr-x 4 root 2000 4.0K Jul 18 09:58 mongodb-mms-automation -rw------- 1 root root 6.4K May 31 11:54 tallylog -rw-rw-r-- 1 root utmp 0 May 31 11:54 wtmp $ ls -ahl /var/log/mongodb-mms-automation total 76M drwxrwsr-x 4 root 2000 4.0K Jul 18 09:58 . drwxr-xr-x 1 root root 4.0K Jul 16 03:19 .. -rw-rw-r-- 1 2000 2000 50 Jul 17 13:42 .slogger-state-automation-agent-verbose.log -rw-rw-r-- 1 2000 2000 50 Jul 17 21:43 .slogger-state-automation-agent.log -rw-rw-r-- 1 2000 2000 48 Jul 17 13:42 .slogger-state-backup-agent.log -rw-rw-r-- 1 2000 2000 50 Jul 17 13:42 .slogger-state-monitoring-agent.log -rw------- 1 2000 2000 1.5K Jul 18 09:58 agent-health-status.json -rw-rw-r-- 1 2000 2000 601 Jul 16 13:42 agent-launcher-script.log -rw-rw-r-- 1 2000 2000 476 Jul 16 13:42 automation-agent-stderr.log -rw-rw-r-- 1 2000 2000 13M Jul 18 09:58 automation-agent-verbose.log -rw-rw-r-- 1 2000 2000 0 Jul 15 20:41 automation-agent-verbose.log.2022-07-15T20-42-00 -rw-rw-r-- 1 2000 2000 1.7K Jul 15 20:42 automation-agent-verbose.log.2022-07-15T20-42-01 -rw-rw-r-- 1 2000 2000 1.1M Jul 16 13:42 automation-agent-verbose.log.2022-07-16T13-42-13 -rw-rw-r-- 1 2000 2000 1.7K Jul 16 13:42 automation-agent-verbose.log.2022-07-16T13-42-14 -rw-rw-r-- 1 2000 2000 15M Jul 17 13:42 automation-agent-verbose.log.2022-07-17T13-42-19 -rw-rw-r-- 1 2000 2000 1.3K Jul 18 01:07 automation-agent.log -rw-rw-r-- 1 2000 2000 886 Jul 15 20:42 automation-agent.log.2022-07-15T20-42-01 -rw-rw-r-- 1 2000 2000 4.1K Jul 15 21:52 automation-agent.log.2022-07-16T13-42-13 -rw-rw-r-- 1 2000 2000 886 Jul 16 13:42 automation-agent.log.2022-07-16T13-42-14 -rw-rw-r-- 1 2000 2000 6.4K Jul 17 21:43 automation-agent.log.2022-07-17T21-43-24 -rw-rw-r-- 1 2000 2000 206K Jul 18 09:58 backup-agent.log -rw-rw-r-- 1 2000 2000 18K Jul 15 22:27 backup-agent.log.2022-07-16T13-42-14 -rw-rw-r-- 1 2000 2000 244K Jul 17 13:42 backup-agent.log.2022-07-17T13-42-50 drwxrwsr-x 2 2000 2000 4.0K Jul 18 09:58 filebeat drwxrws--- 2 root 2000 16K Jul 15 20:40 lost+found -rw------- 1 2000 2000 3.7M Jul 18 09:58 mongodb.log -rw------- 1 2000 2000 36M Jul 18 09:30 mongodb.log.2022-07-18T09-30-43 -rw-rw-r-- 1 2000 2000 349K Jul 18 09:57 monitoring-agent.log -rw-rw-r-- 1 2000 2000 91K Jul 15 22:27 monitoring-agent.log.2022-07-16T13-42-14 -rw-rw-r-- 1 2000 2000 408K Jul 17 13:42 monitoring-agent.log.2022-07-17T13-42-21 -rw-rw-r-- 1 2000 2000 6.7M Jul 18 09:58 readiness.log ```

The replicaset was originally created using the following .yaml:

File

``` apiVersion: mongodb.com/v1 kind: MongoDB metadata: name: mongodb-replset spec: members: 3 version: "4.4.11-ent" service: mongodb-service opsManager: configMapRef: name: mongodb-configmap-proj-gdev01 credentials: mongodb-apisec-proj-gdev01 persistent: true type: ReplicaSet podSpec: cpu: "0.25" memory: "512M" persistence: multiple: data: storage: "2Gi" journal: storage: "500M" security: tls: enabled: false authentication: enabled: true modes: ["SCRAM"] connectivity: replicaSetHorizon: - horizon0: node1:32357 - horizon0: node2:32030 - horizon0: node3:30235 ```

Is adding the initContainer as suggested in https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/config/samples/arbitrary_statefulset_configuration/mongodb.com_v1_hostpath.yaml the recommended way to solve this? I wonder because the example only contains chown commands for /data, nothing for /var/log.

The following MongoDBCommunity works.

---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: mongodb
spec:
  members: 1
  type: ReplicaSet
  version: "4.2.6"
  security:
    authentication:
      modes: ["SCRAM", "SCRAM-SHA-1"]
  users:
    - name: theadmin
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: theadmin-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
        - name: backup
          db: admin
        - name: dbAdminAnyDatabase
          db: admin
        - name: readWriteAnyDatabase
          db: admin
        - name: restore
          db: admin
      scramCredentialsSecretName: theadmin
  additionalMongodConfig:
    storage.wiredTiger.engineConfig.journalCompressor: zlib
# This statefulSet spec is merged with the operator provided statefulSet
  statefulSet:
    spec:
      template:
        spec:
          initContainers:
            - command:
                - chown
                - -R
                - "2000"
                - /data
                - /logs
              image: busybox
              volumeMounts:
              - mountPath: /data
                name: data-volume
              - mountPath: /logs
                name: logs-volume
              securityContext:
                runAsNonRoot: false
                runAsUser: 0
                runAsGroup: 0
              name: change-dir-permissions
          containers:
            - name: mongodb-agent
              readinessProbe:
                failureThreshold: 5000
                initialDelaySeconds: 30

          #required to permit Kustomize to pass on the dynamic Kustomization ServiceAccount name
          serviceAccountName: mongodb-database
          serviceAccount: mongodb-database
          affinity:
            # nodeAffinity:
            #   requiredDuringSchedulingIgnoredDuringExecution:
            #     nodeSelectorTerms:
            #     - matchExpressions:
            #       - key: openebs.io/engine
            #         operator: In
            #         values:
            #         - mayastor
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - mongodb-svc
                topologyKey: "kubernetes.io/hostname"
# Assumption: volumeClaimTemplates with metadata.name = data-volume is used by the
# operator provided mongod containers!
      volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes: [ "ReadWriteOnce" ]
            resources:
              requests:
                storage: 5Gi
        - metadata:
            name: logs-volume
          spec:
            accessModes: [ "ReadWriteOnce" ]
            resources:
              requests:
                storage: 1Gi

Addition:

initContainers:
            - command:
                - chown
                - -R
                - "2000"
                - /data
                - /logs
              image: busybox
              volumeMounts:
              - mountPath: /data
                name: data-volume
              - mountPath: /logs
                name: logs-volume
              securityContext:
                runAsNonRoot: false
                runAsUser: 0
                runAsGroup: 0
              name: change-dir-permissions

mongodb / mongodb-kubernetes-operator