search

ibm-messaging / mq-container

Container images for IBM® MQ
Apache License 2.0
255 stars 189 forks source link

Permission denied on filesystem - MQ 9.3.0.0-r1 MultiInstance - IBM Cloud ROKS 4.10 #510

Closed nicolasmontesi closed 2 years ago

nicolasmontesi commented 2 years ago

I'm trying to deploy MQ 9.3.0.0-r1 in MultiInstance mode but when pod starts I receiving the following error:

Error 71 creating queue manager: AMQ6239E: Permission denied attempting to access filesystem location
'/mnt/mqm/data/mqs.ini'.
AMQ7062E: Permission denied attempting to access an INI file.

I follow this official guide to configure the storage in the correct way: link

My environment:

My QueueManager instance:

apiVersion: mq.ibm.com/v1beta1
kind: QueueManager
metadata:
  name: test
  namespace:test
spec:
  license:
    accept: true
    license: L-RJON-CD3JKX
    use: Production
  pki:
    keys:
      - name: servercert
        secret:
          items:
            - personalcert.key
            - personalcert.crt
          secretName: mq-personal-cert
    trust:
      - name: intermediate
        secret:
          items:
            - ca.crt
          secretName: mq-ca-signcert
  queueManager:
    metrics:
      enabled: false
    availability:
      type: MultiInstance
    name: QM_PROD
    storage:
      defaultClass: ibmc-file-gold-gid
      persistedData:
        class: ibmc-file-gold-gid
        enabled: true
        size: 20Gi
        type: persistent-claim
      queueManager:
        class: ibmc-file-gold-gid
        size: 20Gi
        type: persistent-claim
      recoveryLogs:
        class: ibmc-file-gold-gid
        enabled: true
        size: 20Gi
        type: persistent-claim
  securityContext:
    initVolumeAsRoot: false
    supplementalGroups:
      - 99
  template:
    pod:
      containers:
        - env:
            - name: MQSNOAUT
              value: 'yes'
            - name: DEBUG
              value: 'true'
          name: qmgr
          resources: {}
  version: 9.3.0.0-r1
  web:
    enabled: true

Persistent volume (automatically provided) in which error occurs:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: ibm.io/ibmc-file
    volume.kubernetes.io/storage-provisioner: ibm.io/ibmc-file
  name: data-test-ibm-mq-0
  namespace: test
  finalizers:
    - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/component: integration
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: operator
    app.kubernetes.io/name: ibm-mq
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  volumeName: pvc
  storageClassName: ibmc-file-gold-gid
  volumeMode: Filesystem
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 20Gi

MQ Pod Logs:

2022-11-11T10:57:03.562Z CPU architecture: amd64
2022-11-11T10:57:03.562Z Linux kernel version: 4.18.0-372.26.1.el8_6.x86_64
2022-11-11T10:57:03.562Z Container runtime: kube
2022-11-11T10:57:03.562Z Base image: Red Hat Enterprise Linux 8.6 (Ootpa)
2022-11-11T10:57:03.563Z Running as user ID 1000650000 with primary group 0, and supplementary groups 99,65531,1000650000
2022-11-11T10:57:03.563Z Capabilities: none
2022-11-11T10:57:03.563Z seccomp enforcing mode: disabled
2022-11-11T10:57:03.563Z Process security attributes: system_u:system_r:container_t:s0:c0,c26
2022-11-11T10:57:03.564Z Detected 'nfs4' volume mounted to /mnt/mqm-log
2022-11-11T10:57:03.564Z Detected 'nfs4' volume mounted to /mnt/mqm
2022-11-11T10:57:03.564Z Detected 'nfs4' volume mounted to /mnt/mqm-data
2022-11-11T10:57:03.564Z Multi-instance queue manager: enabled
2022-11-11T10:57:03.564Z Integration-Image created: 2022-06-16T10:35:04+00:00
2022-11-11T10:57:03.564Z Integration-Image tag: ibm-mqadvanced-server-integration:9.3.0.0-r1.20220616103034.e5ef27d-amd64
2022-11-11T10:57:03.582Z Open Tracing is disabled
2022-11-11T10:57:03.613Z Using queue manager name: QM_PROD
2022-11-11T10:57:03.648Z DEBUG: --- Start Diagnostics ---
2022-11-11T10:57:03.660Z DEBUG: /mnt/:
total 12
drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm
drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-data
drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-log

2022-11-11T10:57:03.671Z DEBUG: /mnt/mqm:
total 4
drwxr-xr-x. 2 1000650000 nobody 4096 Nov 11 10:57 data

2022-11-11T10:57:03.676Z DEBUG: /mnt/mqm/data:
total 0

2022-11-11T10:57:03.682Z DEBUG: /mnt/mqm-log/log:
total 0

2022-11-11T10:57:03.688Z DEBUG: /mnt/mqm-data/qmgrs:
total 0

2022-11-11T10:57:03.694Z DEBUG: /var/mqm:
lrwxrwxrwx. 1 root root 13 Jun 15 13:18 /var/mqm -> /mnt/mqm/data

2022-11-11T10:57:03.703Z DEBUG: /var/mqm/errors:
ls: cannot access '/var/mqm/errors': No such file or directory

2022-11-11T10:57:03.719Z DEBUG: /etc/mqm:
total 24
-rw-r--r--. 1 1001 root  745 Jun 15 13:07 15-tls.mqsc.tpl
drwxr-sr-x. 1 1001 root 4096 Jun 16 10:35 MQOpenTracing
-rw-r--r--. 1 1001 root  591 Jun 15 13:07 native-ha.ini.tpl
drwxr-sr-x. 4 root root 4096 Nov 11 10:57 pki
drwxrwsr-x. 1 1001 root 4096 Jun 15 13:19 web

2022-11-11T10:57:03.719Z DEBUG: ffstsummary:

2022-11-11T10:57:03.719Z DEBUG: ---  End Diagnostics  ---
2022-11-11T10:57:11.706Z Warning creating directory structure: 

2022-11-11T10:57:11.706Z Created directory structure under /var/mqm
2022-11-11T10:57:11.706Z DEBUG: --- Start Diagnostics ---
2022-11-11T10:57:11.718Z DEBUG: /mnt/:
total 12
drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm
drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-data
drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-log

2022-11-11T10:57:11.756Z DEBUG: /mnt/mqm:
total 4
drwxrwsr-x. 13 1000650000 nobody 4096 Nov 11 10:57 data

2022-11-11T10:57:11.804Z DEBUG: /mnt/mqm/data:
total 52
drwxrwsr-x. 2 1000650000 nobody 4096 Nov 11 10:57 config
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 conv
drwxrwsrwx. 2 1000650000 nobody 4096 Nov 11 10:57 errors
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 exits
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 exits64
drwxrwsr-x. 2 1000650000 nobody 4096 Nov 11 10:57 log
-rw-rw-r--. 1 1000650000 nobody  571 Nov 11 10:57 mqclient.ini
drwxrwsr-x. 5 1000650000 nobody 4096 Nov 11 10:57 mqft
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 qmgrs
-rw-rw-r--. 1 1000650000 nobody 1941 Nov 11 10:57 service.env
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 sockets
drwxrwsrwx. 2 1000650000 nobody 4096 Nov 11 10:57 trace
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 web

2022-11-11T10:57:11.812Z DEBUG: /mnt/mqm-log/log:
total 0

2022-11-11T10:57:11.815Z DEBUG: /mnt/mqm-data/qmgrs:
total 0

2022-11-11T10:57:11.823Z DEBUG: /var/mqm:
lrwxrwxrwx. 1 root root 13 Jun 15 13:18 /var/mqm -> /mnt/mqm/data

2022-11-11T10:57:11.853Z DEBUG: /var/mqm/errors:
total 100
-rw-rw----. 1 1000650000 nobody 89309 Nov 11 10:57 AMQ28.0.FDC
-rw-rw-r--. 1 1000650000 nobody  2735 Nov 11 10:57 AMQERR01.LOG
-rw-rw-r--. 1 1000650000 nobody  1808 Nov 11 10:57 AMQERR01.json
-rw-rw-r--. 1 1000650000 nobody     0 Nov 11 10:57 AMQERR02.LOG
-rw-rw-r--. 1 1000650000 nobody     0 Nov 11 10:57 AMQERR03.LOG

2022-11-11T10:57:11.863Z DEBUG: /etc/mqm:
total 24
-rw-r--r--. 1 1001 root  745 Jun 15 13:07 15-tls.mqsc.tpl
drwxr-sr-x. 1 1001 root 4096 Jun 16 10:35 MQOpenTracing
-rw-r--r--. 1 1001 root  591 Jun 15 13:07 native-ha.ini.tpl
drwxr-sr-x. 4 root root 4096 Nov 11 10:57 pki
drwxrwsr-x. 1 1001 root 4096 Jun 15 13:19 web

2022-11-11T10:57:12.431Z DEBUG: ffstsummary:
 AMQ28.0.FDC 2022/11/11 10:57:11.302805 Installation1 crtmqdir 28 1 XY019009 xufOpenIniEdit xecU_W_INI_ACCESS_DENIED OK

2022-11-11T10:57:12.431Z DEBUG: ---  End Diagnostics  ---
2022-11-11T10:57:12.431Z Image created: 2022-06-15T13:16:28+00:00
2022-11-11T10:57:12.431Z Image tag: ibm-mqadvanced-server:9.3.0.0-r1.20220615130653.3111d48-amd64
2022-11-11T10:57:13.335Z MQ version: 9.3.0.0
2022-11-11T10:57:13.336Z MQ level: p930-L220606
2022-11-11T10:57:13.336Z MQ license: Production
2022-11-11T10:57:16.429Z Creating queue manager QM_PROD
2022-11-11T10:57:16.429Z Starting web server
2022-11-11T10:57:19.083Z Error 71 creating queue manager: AMQ6239E: Permission denied attempting to access filesystem location
'/mnt/mqm/data/mqs.ini'.
AMQ7062E: Permission denied attempting to access an INI file.

2022-11-11T10:57:19.083Z DEBUG: Writing termination message: /opt/mqm/bin/crtmqm: exit status 71
2022-11-11T10:57:19.083Z /opt/mqm/bin/crtmqm: exit status 71
2022-11-11T10:57:19.083Z DEBUG: --- Start Diagnostics ---
2022-11-11T10:57:19.093Z DEBUG: /mnt/:
total 12
drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm
drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-data
drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-log

2022-11-11T10:57:19.132Z DEBUG: /mnt/mqm:
total 4
drwxrwsr-x. 13 1000650000 nobody 4096 Nov 11 10:57 data

2022-11-11T10:57:19.237Z DEBUG: /mnt/mqm/data:
total 52
drwxrwsr-x. 2 1000650000 nobody 4096 Nov 11 10:57 config
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 conv
drwxrwsrwx. 2 1000650000 nobody 4096 Nov 11 10:57 errors
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 exits
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 exits64
drwxrwsr-x. 2 1000650000 nobody 4096 Nov 11 10:57 log
-rw-rw-r--. 1 1000650000 nobody  571 Nov 11 10:57 mqclient.ini
drwxrwsr-x. 5 1000650000 nobody 4096 Nov 11 10:57 mqft
-rw-rw-r--. 1 1000650000 nobody    0 Nov 11 10:57 mqs.ini
-rw-rw-r--. 1 1000650000 nobody    0 Nov 11 10:57 mqs.ini.tmp
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 qmgrs
-rw-rw-r--. 1 1000650000 nobody 1941 Nov 11 10:57 service.env
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 sockets
drwxrwsrwx. 2 1000650000 nobody 4096 Nov 11 10:57 trace
drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 web

2022-11-11T10:57:19.250Z DEBUG: /mnt/mqm-log/log:
total 0

2022-11-11T10:57:19.254Z DEBUG: /mnt/mqm-data/qmgrs:
total 0

2022-11-11T10:57:19.261Z DEBUG: /var/mqm:
lrwxrwxrwx. 1 root root 13 Jun 15 13:18 /var/mqm -> /mnt/mqm/data

2022-11-11T10:57:19.508Z DEBUG: /var/mqm/errors:
total 100
-rw-rw----. 1 1000650000 nobody 89309 Nov 11 10:57 AMQ28.0.FDC
-rw-rw-r--. 1 1000650000 nobody  2735 Nov 11 10:57 AMQERR01.LOG
-rw-rw-r--. 1 1000650000 nobody  1808 Nov 11 10:57 AMQERR01.json
-rw-rw-r--. 1 1000650000 nobody     0 Nov 11 10:57 AMQERR02.LOG
-rw-rw-r--. 1 1000650000 nobody     0 Nov 11 10:57 AMQERR03.LOG

2022-11-11T10:57:19.518Z DEBUG: /etc/mqm:
total 32
-rw-r-----. 1 1000650000 root  742 Nov 11 10:57 15-tls.mqsc
-rw-r--r--. 1       1001 root  745 Jun 15 13:07 15-tls.mqsc.tpl
drwxr-sr-x. 1       1001 root 4096 Jun 16 10:35 MQOpenTracing
-rw-r--r--. 1       1001 root  591 Jun 15 13:07 native-ha.ini.tpl
drwxr-sr-x. 4 root       root 4096 Nov 11 10:57 pki
drwxrwsr-x. 1       1001 root 4096 Jun 15 13:19 web

2022-11-11T10:57:20.232Z DEBUG: ffstsummary:
 AMQ28.0.FDC 2022/11/11 10:57:11.302805 Installation1 crtmqdir 28 1 XY019009 xufOpenIniEdit xecU_W_INI_ACCESS_DENIED OK

2022-11-11T10:57:20.232Z DEBUG: ---  End Diagnostics  ---
nicolasmontesi commented 2 years ago

Adding GID 65534 to the supplementalGroups list resolve the problem. The worker nodes have RHEL 8 as underlying OS. According to Red Hat changelog:

The nobody user replaces nfsnobody

In Red Hat Enterprise Linux 7, there was:

-     the nobody user and group pair with the ID of 99, and
-     the nfsnobody user and group pair with the ID of 65534, which is the default kernel overflow ID, too. 

Both of these have been merged into the nobody user and group pair, which uses the 65534 ID in Red Hat Enterprise Linux 8. New installations no longer create the nfsnobody pair.

This change reduces the confusion about files that are owned by nobody but have nothing to do with NFS.