search

artemiscloud / activemq-artemis-operator

Apache License 2.0
70 stars 63 forks source link

Unable to create a persistent broker due to "Node Manager can not open file /opt/artemis-broker/data/journal/server.lock" issue #1056

Closed alexandre-touret closed 22 hours ago

alexandre-touret commented 4 days ago

Describe the bug I can't start create a broker enabling the persistence.

I always have the following error message during the startup of the pod:

OTE: Picked up JDK_JAVA_OPTIONS: -Dbroker.properties=/amq/extra/secrets/artemis-broker-props/broker.properties
     _        _               _
    / \  ____| |_  ___ __  __(_) _____
   / _ \|  _ \ __|/ _ \  \/  | |/  __/
  / ___ \ | \/ |_/  __/ |\/| | |\___ \
 /_/   \_\|   \__\____|_|  |_|_|/___ /
 Apache ActiveMQ Artemis 2.38.0

2024-11-22 14:23:43,462 INFO  [org.apache.activemq.artemis.core.server] AMQ221082: Initializing metrics plugin com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin with properties: {}
2024-11-22 14:23:43,822 INFO  [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server version 2.38.0
2024-11-22 14:23:44,380 WARN  [org.apache.activemq.artemis.core.server] AMQ222141: Node Manager can not open file /opt/artemis-broker/data/journal/server.lock
java.io.IOException: No such file or directory
        at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method) ~[?:?]
        at java.base/java.io.File.createNewFile(File.java:1043) ~[?:?]
        at org.apache.activemq.artemis.core.server.impl.FileBasedNodeManager.setUpServerLockFile(FileBasedNodeManager.java:145) [artemis-server-2.38.0.jar:2.38.0]
        at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.setUpServerLockFile(FileLockNodeManager.java:112) [artemis-server-2.38.0.jar:2.38.0]
        at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.start(FileLockNodeManager.java:103) [artemis-server-2.38.0.jar:2.38.0]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:718) [artemis-server-2.38.0.jar:2.38.0]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:632) [artemis-server-2.38.0.jar:2.38.0]
        at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66) [artemis-cli-2.38.0.jar:2.38.0]
        at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130) [artemis-cli-2.38.0.jar:2.38.0]
        at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:222) [artemis-cli-2.38.0.jar:2.38.0]
        at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:168) [artemis-cli-2.38.0.jar:2.38.0]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
        at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:152) [artemis-boot.jar:2.38.0]
        at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64) [artemis-boot.jar:2.38.0]
2024-11-22 14:23:44,396 ERROR [org.apache.activemq.artemis.core.server] AMQ224097: Failed to start server

Here is my stack:

Below, the steps I applied:

kubectl create namespace activemq-artemis-operator
kubectl config set-context --current --namespace activemq-artemis-operator
 ./deploy/install_opr.sh
 kubectl apply -f - <<EOF
apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: artemis-broker
spec:
  console:
    expose: true
  deploymentPlan:
    size: 2
    persistenceEnabled: true
    messageMigration: true
    enableMetricsPlugin: true
EOF

You can get below the events:

LAST SEEN   TYPE      REASON                   OBJECT                                                      MESSAGE
3m29s       Normal    Scheduled                pod/activemq-artemis-controller-manager-7f57ccb5b4-864f9    Successfully assigned activemq-artemis-operator/activemq-artemis-controller-manager-7f57ccb5b4-864f9 to gke-gke-cluster-dev-node-pool20241010-2b30ce33-ia5o
3m29s       Normal    SuccessfulCreate         replicaset/activemq-artemis-controller-manager-7f57ccb5b4   Created pod: activemq-artemis-controller-manager-7f57ccb5b4-864f9
3m29s       Normal    ScalingReplicaSet        deployment/activemq-artemis-controller-manager              Scaled up replica set activemq-artemis-controller-manager-7f57ccb5b4 to 1
3m28s       Normal    Pulled                   pod/activemq-artemis-controller-manager-7f57ccb5b4-864f9    Container image "quay.io/artemiscloud/activemq-artemis-operator:1.2.7" already present on machine
3m28s       Normal    Created                  pod/activemq-artemis-controller-manager-7f57ccb5b4-864f9    Created container manager
3m28s       Normal    Started                  pod/activemq-artemis-controller-manager-7f57ccb5b4-864f9    Started container manager
3m26s       Normal    LeaderElection           lease/d864aab0.amq.io                                       activemq-artemis-controller-manager-7f57ccb5b4-864f9_5dbcf866-cc81-406a-bbd6-17fde829ae5e became leader
2m29s       Normal    WaitForFirstConsumer     persistentvolumeclaim/artemis-broker-artemis-broker-ss-0    waiting for first consumer to be created before binding
2m29s       Normal    ADD                      service/artemis-broker-hdls-svc                             activemq-artemis-operator/artemis-broker-hdls-svc
2m29s       Normal    ADD                      service/artemis-broker-ping-svc                             activemq-artemis-operator/artemis-broker-ping-svc
2m29s       Normal    SuccessfulCreate         statefulset/artemis-broker-ss                               create Claim artemis-broker-artemis-broker-ss-0 Pod artemis-broker-ss-0 in StatefulSet artemis-broker-ss success
2m28s       Normal    Provisioning             persistentvolumeclaim/artemis-broker-artemis-broker-ss-0    External provisioner is provisioning volume for claim "activemq-artemis-operator/artemis-broker-artemis-broker-ss-0"
2m28s       Normal    ExternalProvisioning     persistentvolumeclaim/artemis-broker-artemis-broker-ss-0    Waiting for a volume to be created either by the external provisioner 'pd.csi.storage.gke.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
2m28s       Normal    SuccessfulCreate         statefulset/artemis-broker-ss                               create Pod artemis-broker-ss-0 in StatefulSet artemis-broker-ss successful
2m24s       Normal    ProvisioningSucceeded    persistentvolumeclaim/artemis-broker-artemis-broker-ss-0    Successfully provisioned volume pvc-1b4438d0-15c8-45af-beec-2f99e10cb1f9
2m23s       Normal    Scheduled                pod/artemis-broker-ss-0                                     Successfully assigned activemq-artemis-operator/artemis-broker-ss-0 to gke-gke-cluster-dev-node-pool20241010-2b30ce33-ia5o
2m16s       Normal    SuccessfulAttachVolume   pod/artemis-broker-ss-0                                     AttachVolume.Attach succeeded for volume "pvc-1b4438d0-15c8-45af-beec-2f99e10cb1f9"
2m14s       Normal    Started                  pod/artemis-broker-ss-0                                     Started container artemis-broker-container-init
2m14s       Normal    Created                  pod/artemis-broker-ss-0                                     Created container artemis-broker-container-init
2m14s       Normal    Pulled                   pod/artemis-broker-ss-0                                     Container image "quay.io/artemiscloud/activemq-artemis-broker-init@sha256:55b614c1f3ff359ae1f98d0de831d649f7f1a3fa0bdb6979aa81dd8fd6156f45" already present on machine
70s         Normal    Created                  pod/artemis-broker-ss-0                                     Created container artemis-broker-container
70s         Normal    Started                  pod/artemis-broker-ss-0                                     Started container artemis-broker-container
35s         Warning   BackOff                  pod/artemis-broker-ss-0                                     Back-off restarting failed container artemis-broker-container in pod artemis-broker-ss-0_activemq-artemis-operator(0dc20359-45f6-47e1-9694-6699ba92f36c)
21s         Normal    Pulled                   pod/artemis-broker-ss-0                                     Container image "quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:1c2d9fcc1d9462a81dce163e74629d294728d4180262c83ccfa908868ff28d26" already present on machine

And the description of the StateFulSet

 kubectl describe sts 
Name:               artemis-broker-ss
Namespace:          activemq-artemis-operator
CreationTimestamp:  Fri, 22 Nov 2024 14:22:52 +0000
Selector:           ActiveMQArtemis=artemis-broker,application=artemis-broker-app
Labels:             ActiveMQArtemis=artemis-broker
                    application=artemis-broker-app
Annotations:        <none>
Replicas:           2 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  ActiveMQArtemis=artemis-broker
           application=artemis-broker-app
  Init Containers:
   artemis-broker-container-init:
    Image:           quay.io/artemiscloud/activemq-artemis-broker-init@sha256:55b614c1f3ff359ae1f98d0de831d649f7f1a3fa0bdb6979aa81dd8fd6156f45
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/bash
    Args:
      -c
      /opt/amq/bin/launch.sh && /opt/amq-broker/script/default.sh
    Environment:
      AMQ_ROLE:                         admin
      AMQ_NAME:                         amq-broker
      AMQ_TRANSPORTS:                   
      AMQ_QUEUES:                       
      AMQ_ADDRESSES:                    
      AMQ_GLOBAL_MAX_SIZE:              100 mb
      AMQ_REQUIRE_LOGIN:                false
      AMQ_EXTRA_ARGS:                   --no-autotune
      AMQ_ANYCAST_PREFIX:               
      AMQ_MULTICAST_PREFIX:             
      POD_NAMESPACE:                    
      AMQ_JOURNAL_TYPE:                 nio
      TRIGGERED_ROLL_COUNT:             0
      PING_SVC_NAME:                    artemis-broker-ping-svc
      OPENSHIFT_DNS_PING_SERVICE_PORT:  7800
      AMQ_DATA_DIR:                     /opt/artemis-broker/data
      AMQ_DATA_DIR_LOGGING:             true
      AMQ_CLUSTERED:                    true
      AMQ_ENABLE_JOLOKIA_AGENT:         false
      AMQ_ENABLE_MANAGEMENT_RBAC:       false
      AMQ_ENABLE_METRICS_PLUGIN:        true
      RUN_BROKER:                       false
      CONFIG_INSTANCE_DIR:              /amq/init/config
      AMQ_CLUSTER_PASSWORD:             <set to the key 'AMQ_CLUSTER_PASSWORD' in secret 'artemis-broker-credentials-secret'>  Optional: false
      AMQ_CLUSTER_USER:                 <set to the key 'AMQ_CLUSTER_USER' in secret 'artemis-broker-credentials-secret'>      Optional: false
      AMQ_PASSWORD:                     <set to the key 'AMQ_PASSWORD' in secret 'artemis-broker-credentials-secret'>          Optional: false
      AMQ_USER:                         <set to the key 'AMQ_USER' in secret 'artemis-broker-credentials-secret'>              Optional: false
      AMQ_ACCEPTORS:                    <set to the key 'AMQ_ACCEPTORS' in secret 'artemis-broker-netty-secret'>               Optional: false
      AMQ_CONNECTORS:                   <set to the key 'AMQ_CONNECTORS' in secret 'artemis-broker-netty-secret'>              Optional: false
    Mounts:
      /amq/extra/secrets/artemis-broker-props from secret-artemis-broker-props (ro)
      /amq/init/config from amq-cfg-dir (rw)
      /init_cfg_root from tool-dir (rw)
  Containers:
   artemis-broker-container:
    Image:           quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:1c2d9fcc1d9462a81dce163e74629d294728d4180262c83ccfa908868ff28d26
    Port:            8161/TCP
    Host Port:       0/TCP
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/bash
      -c
      export STATEFUL_SET_ORDINAL=${HOSTNAME##*-}; export JDK_JAVA_OPTIONS=${JDK_JAVA_OPTIONS//\$\{STATEFUL_SET_ORDINAL\}/${HOSTNAME##*-}}; export FQ_HOST_NAME=$(hostname -f); export JAVA_ARGS_APPEND=$( echo ${JAVA_ARGS_APPEND} | sed "s/FQ_HOST_NAME/${FQ_HOST_NAME}/"); exec /opt/amq/bin/launch.sh
      start
    Liveness:   tcp-socket :8161 delay=5s timeout=5s period=10s #success=1 #failure=3
    Readiness:  exec [/bin/bash -c /opt/amq/bin/readinessProbe.sh 1] delay=5s timeout=5s period=10s #success=1 #failure=3
    Environment:
      AMQ_ROLE:                         admin
      AMQ_NAME:                         amq-broker
      AMQ_TRANSPORTS:                   
      AMQ_QUEUES:                       
      AMQ_ADDRESSES:                    
      AMQ_GLOBAL_MAX_SIZE:              100 mb
      AMQ_REQUIRE_LOGIN:                false
      AMQ_EXTRA_ARGS:                   --no-autotune
      AMQ_ANYCAST_PREFIX:               
      AMQ_MULTICAST_PREFIX:             
      POD_NAMESPACE:                    
      AMQ_JOURNAL_TYPE:                 nio
      TRIGGERED_ROLL_COUNT:             79304a5e
      PING_SVC_NAME:                    artemis-broker-ping-svc
      OPENSHIFT_DNS_PING_SERVICE_PORT:  7800
      AMQ_DATA_DIR:                     /opt/artemis-broker/data
      AMQ_DATA_DIR_LOGGING:             true
      AMQ_CLUSTERED:                    true
      AMQ_ENABLE_JOLOKIA_AGENT:         false
      AMQ_ENABLE_MANAGEMENT_RBAC:       false
      AMQ_ENABLE_METRICS_PLUGIN:        true
      CONFIG_BROKER:                    false
      CONFIG_INSTANCE_DIR:              /amq/init/config
      JDK_JAVA_OPTIONS:                 -Dbroker.properties=/amq/extra/secrets/artemis-broker-props/broker.properties
      AMQ_CLUSTER_PASSWORD:             <set to the key 'AMQ_CLUSTER_PASSWORD' in secret 'artemis-broker-credentials-secret'>  Optional: false
      AMQ_CLUSTER_USER:                 <set to the key 'AMQ_CLUSTER_USER' in secret 'artemis-broker-credentials-secret'>      Optional: false
      AMQ_PASSWORD:                     <set to the key 'AMQ_PASSWORD' in secret 'artemis-broker-credentials-secret'>          Optional: false
      AMQ_USER:                         <set to the key 'AMQ_USER' in secret 'artemis-broker-credentials-secret'>              Optional: false
      AMQ_ACCEPTORS:                    <set to the key 'AMQ_ACCEPTORS' in secret 'artemis-broker-netty-secret'>               Optional: false
      AMQ_CONNECTORS:                   <set to the key 'AMQ_CONNECTORS' in secret 'artemis-broker-netty-secret'>              Optional: false
    Mounts:
      /amq/extra/secrets/artemis-broker-props from secret-artemis-broker-props (ro)
      /amq/init/config from amq-cfg-dir (rw)
      /opt/artemis-broker/data from artemis-broker (rw)
  Volumes:
   artemis-broker:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  artemis-broker
    ReadOnly:   false
   secret-artemis-broker-props:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  artemis-broker-props
    Optional:    false
   amq-cfg-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   tool-dir:
    Type:          EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:        
    SizeLimit:     <unset>
  Node-Selectors:  <none>
  Tolerations:     <none>
Volume Claims:
  Name:          artemis-broker
  StorageClass:  
  Labels:        ActiveMQArtemis=artemis-broker
                 application=artemis-broker-app
  Annotations:   <none>
  Capacity:      2Gi
  Access Modes:  [ReadWriteOnce]
Events:
  Type    Reason            Age   From                    Message
  ----    ------            ----  ----                    -------
  Normal  SuccessfulCreate  11m   statefulset-controller  create Claim artemis-broker-artemis-broker-ss-0 Pod artemis-broker-ss-0 in StatefulSet artemis-broker-ss success
  Normal  SuccessfulCreate  11m   statefulset-controller  create Pod artemis-broker-ss-0 in StatefulSet artemis-broker-ss successful

And the description of the PVC

kubectl get pvc
NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
artemis-broker-artemis-broker-ss-0   Bound    pvc-1b4438d0-15c8-45af-beec-2f99e10cb1f9   2Gi        RWO            standard-rwo   <unset>                 13m
kubectl describe pvc artemis-broker-artemis-broker-ss-0
Name:          artemis-broker-artemis-broker-ss-0
Namespace:     activemq-artemis-operator
StorageClass:  standard-rwo
Status:        Bound
Volume:        pvc-1b4438d0-15c8-45af-beec-2f99e10cb1f9
Labels:        ActiveMQArtemis=artemis-broker
               application=artemis-broker-app
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
               volume.kubernetes.io/selected-node: gke-gke-cluster-dev-node-pool20241010-2b30ce33-ia5o
               volume.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       artemis-broker-ss-0
Events:
  Type    Reason                 Age                From                                                                                              Message
  ----    ------                 ----               ----                                                                                              -------
  Normal  WaitForFirstConsumer   14m                persistentvolume-controller                                                                       waiting for first consumer to be created before binding
  Normal  ExternalProvisioning   14m (x2 over 14m)  persistentvolume-controller                                                                       Waiting for a volume to be created either by the external provisioner 'pd.csi.storage.gke.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  Provisioning           14m                pd.csi.storage.gke.io_gke-faf632fb6bce41508713-f30e-6f02-vm_a6418400-f161-45ce-8978-b7b1e0ea39a9  External provisioner is provisioning volume for claim "activemq-artemis-operator/artemis-broker-artemis-broker-ss-0"
  Normal  ProvisioningSucceeded  13m                pd.csi.storage.gke.io_gke-faf632fb6bce41508713-f30e-6f02-vm_a6418400-f161-45ce-8978-b7b1e0ea39a9  Successfully provisioned volume pvc-1b4438d0-15c8-45af-beec-2f99e10cb1f9

How can I fix this issue please ?

[!TIP] Vote this issue reacting with :+1: or :-1:

brusdev commented 4 days ago

I suspect this issue could be cause by a custom security context of your Kubernetes cluster. Other users with a similar issue fixed it the field spec.deploymentPlan.podSecurityContext.fsGroup to 0, for further details see https://github.com/artemiscloud/activemq-artemis-operator/issues/187#issuecomment-2045575143

Could you dump and share your ActiveMQArtemis CR before trying to set the field spec.deploymentPlan.podSecurityContext.fsGroup to 0?

kubectl get ActiveMQArtemis <NAME> -o yaml
alexandre-touret commented 4 days ago

Thanks for your response. Here the output of the command you asked me

kubectl get ActiveMQArtemis artemis-broker -o yaml
apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"broker.amq.io/v1beta1","kind":"ActiveMQArtemis","metadata":{"annotations":{},"name":"artemis-broker","namespace":"activemq-artemis-operator"},"spec":{"deploymentPlan":{"enableMetricsPlugin":true,"messageMigration":true,"persistenceEnabled":true,"size":2}}}
  creationTimestamp: "2024-11-22T14:55:27Z"
  generation: 1
  name: artemis-broker
  namespace: activemq-artemis-operator
  resourceVersion: "48789251"
  uid: 5bce607f-6d1b-498c-b388-8f7768ba0eca
spec:
  deploymentPlan:
    enableMetricsPlugin: true
    messageMigration: true
    persistenceEnabled: true
    size: 2
status:
  conditions:
  - lastTransitionTime: "2024-11-22T14:55:27Z"
    message: ""
    observedGeneration: 1
    reason: ValidationSucceded
    status: "True"
    type: Valid
  - lastTransitionTime: "2024-11-22T14:55:27Z"
    message: no available brokers from deployed condition
    reason: UnableToRetrieveStatus
    status: Unknown
    type: BrokerPropertiesApplied
  - lastTransitionTime: "2024-11-22T14:55:27Z"
    message: '0/2 pods ready {artemis-broker-ss-0: Pending [{PodReadyToStartContainers=False}{Initialized=False
      ContainersNotInitialized containers with incomplete status: [artemis-broker-container-init]}{Ready=False
      ContainersNotReady containers with unready status: [artemis-broker-container]}{ContainersReady=False
      ContainersNotReady containers with unready status: [artemis-broker-container]}{PodScheduled=True}]}'
    reason: PodsNotReady
    status: "False"
    type: Deployed
  - lastTransitionTime: "2024-11-22T14:55:27Z"
    message: Some conditions are not met
    reason: WaitingForAllConditions
    status: "False"
    type: Ready
  deploymentPlanSize: 2
  podStatus:
    starting:
    - artemis-broker-ss-0
  scaleLabelSelector: ActiveMQArtemis=artemis-broker,application=artemis-broker-app
  upgrade:
    majorUpdates: true
    minorUpdates: true
    patchUpdates: true
    securityUpdates: true
  version:
    brokerVersion: 2.38.0
    image: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:1c2d9fcc1d9462a81dce163e74629d294728d4180262c83ccfa908868ff28d26
    initImage: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:55b614c1f3ff359ae1f98d0de831d649f7f1a3fa0bdb6979aa81dd8fd6156f45
brusdev commented 3 days ago

@alexandre-touret thanks for sharing your AcriveMQArtemis CR. Could you also share the dump of your broker pod before setting the field spec.deploymentPlan.podSecurityContext.fsGroup to 0?

kubectl get pod artemis-broker-ss-0 -o yaml
alexandre-touret commented 3 days ago

Hi You can find below the output of the command.

BTW I applied the configuration you mentioned spec.deploymentPlan.podSecurityContext.fsGroup to 0. It works

kubectl get pod artemis-broker-cluster-ss-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-11-22T15:33:08Z"
  generateName: artemis-broker-cluster-ss-
  labels:
    ActiveMQArtemis: artemis-broker-cluster
    application: artemis-broker-cluster-app
    apps.kubernetes.io/pod-index: "0"
    controller-revision-hash: artemis-broker-cluster-ss-7cb6d84fb6
    statefulset.kubernetes.io/pod-name: artemis-broker-cluster-ss-0
  name: artemis-broker-cluster-ss-0
  namespace: activemq-artemis-operator
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: artemis-broker-cluster-ss
    uid: 0e9d9cb0-e8ae-4433-b4fe-e1ecaac7b1c5
  resourceVersion: "48819710"
  uid: 89529094-5873-4559-9e33-93c9f29da5fc
spec:
  affinity: {}
  automountServiceAccountToken: true
  containers:
  - command:
    - /bin/bash
    - -c
    - export STATEFUL_SET_ORDINAL=${HOSTNAME##*-}; export JDK_JAVA_OPTIONS=${JDK_JAVA_OPTIONS//\$\{STATEFUL_SET_ORDINAL\}/${HOSTNAME##*-}};
      export FQ_HOST_NAME=$(hostname -f); export JAVA_ARGS_APPEND=$( echo ${JAVA_ARGS_APPEND}
      | sed "s/FQ_HOST_NAME/${FQ_HOST_NAME}/"); exec /opt/amq/bin/launch.sh
    - start
    env:
    - name: AMQ_ROLE
      value: admin
    - name: AMQ_NAME
      value: amq-broker
    - name: AMQ_TRANSPORTS
    - name: AMQ_QUEUES
    - name: AMQ_ADDRESSES
    - name: AMQ_GLOBAL_MAX_SIZE
      value: 100 mb
    - name: AMQ_REQUIRE_LOGIN
      value: "false"
    - name: AMQ_EXTRA_ARGS
      value: --no-autotune
    - name: AMQ_ANYCAST_PREFIX
    - name: AMQ_MULTICAST_PREFIX
    - name: POD_NAMESPACE
    - name: AMQ_JOURNAL_TYPE
      value: nio
    - name: TRIGGERED_ROLL_COUNT
      value: 79304a5e
    - name: PING_SVC_NAME
      value: artemis-broker-cluster-ping-svc
    - name: OPENSHIFT_DNS_PING_SERVICE_PORT
      value: "7800"
    - name: AMQ_DATA_DIR
      value: /opt/artemis-broker-cluster/data
    - name: AMQ_DATA_DIR_LOGGING
      value: "true"
    - name: AMQ_CLUSTERED
      value: "true"
    - name: AMQ_ENABLE_JOLOKIA_AGENT
      value: "false"
    - name: AMQ_ENABLE_MANAGEMENT_RBAC
      value: "false"
    - name: AMQ_ENABLE_METRICS_PLUGIN
      value: "true"
    - name: CONFIG_BROKER
      value: "false"
    - name: CONFIG_INSTANCE_DIR
      value: /amq/init/config
    - name: JDK_JAVA_OPTIONS
      value: -Dbroker.properties=/amq/extra/secrets/artemis-broker-cluster-props/broker.properties
    - name: AMQ_CLUSTER_PASSWORD
      valueFrom:
        secretKeyRef:
          key: AMQ_CLUSTER_PASSWORD
          name: artemis-broker-cluster-credentials-secret
    - name: AMQ_CLUSTER_USER
      valueFrom:
        secretKeyRef:
          key: AMQ_CLUSTER_USER
          name: artemis-broker-cluster-credentials-secret
    - name: AMQ_PASSWORD
      valueFrom:
        secretKeyRef:
          key: AMQ_PASSWORD
          name: artemis-broker-cluster-credentials-secret
    - name: AMQ_USER
      valueFrom:
        secretKeyRef:
          key: AMQ_USER
          name: artemis-broker-cluster-credentials-secret
    - name: AMQ_ACCEPTORS
      valueFrom:
        secretKeyRef:
          key: AMQ_ACCEPTORS
          name: artemis-broker-cluster-netty-secret
    - name: AMQ_CONNECTORS
      valueFrom:
        secretKeyRef:
          key: AMQ_CONNECTORS
          name: artemis-broker-cluster-netty-secret
    image: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:1c2d9fcc1d9462a81dce163e74629d294728d4180262c83ccfa908868ff28d26
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      tcpSocket:
        port: 8161
      timeoutSeconds: 5
    name: artemis-broker-cluster-container
    ports:
    - containerPort: 8161
      name: wconsj
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - /bin/bash
        - -c
        - /opt/amq/bin/readinessProbe.sh 1
      failureThreshold: 3
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: false
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/artemis-broker-cluster/data
      name: artemis-broker-cluster
    - mountPath: /amq/extra/secrets/artemis-broker-cluster-props
      name: secret-artemis-broker-cluster-props
      readOnly: true
    - mountPath: /amq/init/config
      name: amq-cfg-dir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-x5qxj
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: artemis-broker-cluster-ss-0
  initContainers:
  - args:
    - -c
    - /opt/amq/bin/launch.sh && /opt/amq-broker/script/default.sh
    command:
    - /bin/bash
    env:
    - name: AMQ_ROLE
      value: admin
    - name: AMQ_NAME
      value: amq-broker
    - name: AMQ_TRANSPORTS
    - name: AMQ_QUEUES
    - name: AMQ_ADDRESSES
    - name: AMQ_GLOBAL_MAX_SIZE
      value: 100 mb
    - name: AMQ_REQUIRE_LOGIN
      value: "false"
    - name: AMQ_EXTRA_ARGS
      value: --no-autotune
    - name: AMQ_ANYCAST_PREFIX
    - name: AMQ_MULTICAST_PREFIX
    - name: POD_NAMESPACE
    - name: AMQ_JOURNAL_TYPE
      value: nio
    - name: TRIGGERED_ROLL_COUNT
      value: "0"
    - name: PING_SVC_NAME
      value: artemis-broker-cluster-ping-svc
    - name: OPENSHIFT_DNS_PING_SERVICE_PORT
      value: "7800"
    - name: AMQ_DATA_DIR
      value: /opt/artemis-broker-cluster/data
    - name: AMQ_DATA_DIR_LOGGING
      value: "true"
    - name: AMQ_CLUSTERED
      value: "true"
    - name: AMQ_ENABLE_JOLOKIA_AGENT
      value: "false"
    - name: AMQ_ENABLE_MANAGEMENT_RBAC
      value: "false"
    - name: AMQ_ENABLE_METRICS_PLUGIN
      value: "true"
    - name: RUN_BROKER
      value: "false"
    - name: CONFIG_INSTANCE_DIR
      value: /amq/init/config
    - name: AMQ_CLUSTER_PASSWORD
      valueFrom:
        secretKeyRef:
          key: AMQ_CLUSTER_PASSWORD
          name: artemis-broker-cluster-credentials-secret
    - name: AMQ_CLUSTER_USER
      valueFrom:
        secretKeyRef:
          key: AMQ_CLUSTER_USER
          name: artemis-broker-cluster-credentials-secret
    - name: AMQ_PASSWORD
      valueFrom:
        secretKeyRef:
          key: AMQ_PASSWORD
          name: artemis-broker-cluster-credentials-secret
    - name: AMQ_USER
      valueFrom:
        secretKeyRef:
          key: AMQ_USER
          name: artemis-broker-cluster-credentials-secret
    - name: AMQ_ACCEPTORS
      valueFrom:
        secretKeyRef:
          key: AMQ_ACCEPTORS
          name: artemis-broker-cluster-netty-secret
    - name: AMQ_CONNECTORS
      valueFrom:
        secretKeyRef:
          key: AMQ_CONNECTORS
          name: artemis-broker-cluster-netty-secret
    image: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:55b614c1f3ff359ae1f98d0de831d649f7f1a3fa0bdb6979aa81dd8fd6156f45
    imagePullPolicy: IfNotPresent
    name: artemis-broker-cluster-container-init
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: false
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /amq/init/config
      name: amq-cfg-dir
    - mountPath: /init_cfg_root
      name: tool-dir
    - mountPath: /amq/extra/secrets/artemis-broker-cluster-props
      name: secret-artemis-broker-cluster-props
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-x5qxj
      readOnly: true
  nodeName: gke-gke-cluster-dev-node-pool20241010-2b30ce33-ia5o
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  serviceAccount: default
  serviceAccountName: default
  subdomain: artemis-broker-cluster-hdls-svc
  terminationGracePeriodSeconds: 60
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: artemis-broker-cluster
    persistentVolumeClaim:
      claimName: artemis-broker-cluster-artemis-broker-cluster-ss-0
  - name: secret-artemis-broker-cluster-props
    secret:
      defaultMode: 420
      secretName: artemis-broker-cluster-props
  - emptyDir: {}
    name: amq-cfg-dir
  - emptyDir: {}
    name: tool-dir
  - name: kube-api-access-x5qxj
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-11-22T15:33:14Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2024-11-22T15:33:17Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-11-22T15:33:33Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-11-22T15:33:33Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-11-22T15:33:08Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://2cbbffcacbceee4a46dabc5c1dd1414d2c8082f3f1172b0c1a896698289eff56
    image: sha256:51e758677edd5f762ca85830d486afbc725bde0bed9b25b5a13269aca8dd72ab
    imageID: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:1c2d9fcc1d9462a81dce163e74629d294728d4180262c83ccfa908868ff28d26
    lastState: {}
    name: artemis-broker-cluster-container
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-11-22T15:33:17Z"
  hostIP: 10.123.150.148
  hostIPs:
  - ip: 10.123.150.148
  initContainerStatuses:
  - containerID: containerd://c9723b4c8edaf7478dfbbaaf2684eafd013e9330233942aa5259e1a622054c32
    image: sha256:f10f348f0100cd4ad9d8e551e79cf9b35e1ae1b0fb02df33ad2cd7c4072a04c9
    imageID: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:55b614c1f3ff359ae1f98d0de831d649f7f1a3fa0bdb6979aa81dd8fd6156f45
    lastState: {}
    name: artemis-broker-cluster-container-init
    ready: true
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://c9723b4c8edaf7478dfbbaaf2684eafd013e9330233942aa5259e1a622054c32
        exitCode: 0
        finishedAt: "2024-11-22T15:33:17Z"
        reason: Completed
        startedAt: "2024-11-22T15:33:14Z"
  phase: Running
  podIP: 100.83.131.33
  podIPs:
  - ip: 100.83.131.33
  qosClass: BestEffort
  startTime: "2024-11-22T15:33:08Z"
brusdev commented 3 days ago

I'm glad that setting spec.deploymentPlan.podSecurityContext.fsGroup to 0 worked. Thanks for sharing your dumps. For some reason the persistent volumes created by your Kubernetes cluster allow write access only to the user of the group 0 but I don't see any custom security context settings in your dumps.

alexandre-touret commented 1 day ago

I don't remember if I captured the configuration before applying the configuration (i.e., fsGroup) or after. Anyway, it could be nice to add this tip to your documentation.

brusdev commented 1 day ago

@alexandre-touret your proposal to add this tip to your documentation makes sense to me but I need to understand what is causing this issue in some Kubernetes clusters and what are the alternatives.

alexandre-touret commented 22 hours ago

IMO we can close this issue Perhaps, I will submit another one for improving the documentation ;)