search

WASdev / websphere-liberty-operator

Deploy and manage containerized Liberty applications on Kubernetes. Documentation: https://ibm.biz/wlo-docs
Apache License 2.0
9 stars 4 forks source link

SVT: Application failed to Deploy on WLO but Deployed successfully on OLO in a zLinux cluster #691

Open abdulmateen-1 opened 1 month ago

abdulmateen-1 commented 1 month ago

We use Websphere Liberty Operator to test new release of Instant-On.

We use the WebSphere Liberty Operator (WLO) to test new releases of Instant-On. While testing Instant-On 24.0.0.10, I encountered an issue where the application pods were not being created. 

Error creating: pods "dt10mq-7d57cf64-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "CHECKPOINT_RESTORE": capability may not be added, provider restricted-
v2: .containers[0].capabilities.add: Invalid value: "SETPCAP": capability may not be added, provider restricted-v2:
 .containers[0].allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed,
 provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by
 user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-
anyuid": Forbidden: not usable by user or serviceaccount, pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/app]: Forbidden: seccomp may not be set,
 provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-
v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or 
serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": 
Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

I was able to deploy the same image successfully using the Open Liberty Operators.

leochr commented 1 month ago

@abdulmateen-1 It was mentioned in Slack that OLO version was 1.3.3 while WLO is 1.4.0. Is that correct?

Please share the following configurations that were being used during the testing of Instant-On application:

  1. Security Context Constraint (SCC) yaml
  2. service account configuration details
  3. securityContext configuration details in WebSphereLibertyApplication custom resource (CR)
abdulmateen-1 commented 1 month ago

Yes that is correct. The WLO version is 1.4.0 and the OLO version is 1.3.3.

Security Context Constraint yaml file

kind: SecurityContextConstraints
metadata:
  annotations:
    kubernetes.io/description: criu-scc is based on the restricted SCC but removes any restrictions
      that prevent the restore image from running successfully.
  generation: 1
  name: cap-cr-scc
allowHostDirVolumePlugin: true
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
allowedCapabilities: null
apiVersion: security.openshift.io/v1
defaultAddCapabilities:
- CHECKPOINT_RESTORE
- SETPCAP
fsGroup:
  type: RunAsAny
  # type: MustRunAs
groups:
- system:authenticated
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities:
- KILL
- MKNOD
- SETUID
- SETGID
runAsUser:
  type: RunAsAny
  # type: MustRunAs
  # uid: 1001
seLinuxContext:
  type: RunAsAny
  # type: MustRunAs
supplementalGroups:
  type: RunAsAny
users: []
volumes:
- configMap
- downwardAPI
- emptyDir
- hostPath
- persistentVolumeClaim
- projected
- secret

Service Account Configuration details

kind: ServiceAccount
apiVersion: v1
metadata:
  name: instanton-sa
  namespace: nest-daytrader10-instanton
  uid: f17137cb-42db-416b-8e9f-ac300c4f3d26
  resourceVersion: '3035767'
  creationTimestamp: '2024-09-27T16:43:49Z'
  annotations:
    argocd.argoproj.io/sync-wave: '5'
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{"argocd.argoproj.io/sync-wave":"5"},"name":"instanton-sa","namespace":"nest-daytrader10-instanton"}}
    openshift.io/internal-registry-pull-secret-ref: instanton-sa-dockercfg-p2mks
  managedFields:
    - manager: openshift.io/image-registry-pull-secrets_service-account-controller
      operation: Apply
      apiVersion: v1
      time: '2024-09-27T16:43:49Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:imagePullSecrets': {}
        'f:metadata':
          'f:annotations':
            'f:openshift.io/internal-registry-pull-secret-ref': {}
        'f:secrets':
          'k:{"name":"instanton-sa-dockercfg-p2mks"}': {}
    - manager: argocd-controller
      operation: Update
      apiVersion: v1
      time: '2024-09-27T16:55:05Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:argocd.argoproj.io/sync-wave': {}
    - manager: kubectl-client-side-apply
      operation: Update
      apiVersion: v1
      time: '2024-09-30T16:26:15Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
secrets:
  - name: instanton-sa-dockercfg-p2mks
imagePullSecrets:
  - name: instanton-sa-dockercfg-p2mks

% oc describe sa instanton-sa
  Name:                instanton-sa
Namespace:           nest-daytrader10-instanton
Labels:              <none>
Annotations:         argocd.argoproj.io/sync-wave: 5
                     openshift.io/internal-registry-pull-secret-ref: instanton-sa-dockercfg-p2mks
Image pull secrets:  instanton-sa-dockercfg-p2mks
Mountable secrets:   instanton-sa-dockercfg-p2mks
Tokens:              <none>
Events:              <none>

We then add the security context constraint to the service account using this command. oc adm policy add-scc-to-user cap-cr-scc -z instanton-sa

Security Context Config in WLO Custom Resource.

% oc get deployment daytrader10-wlo-instanton -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2024-09-30T20:03:55Z"
  generation: 1
  labels:
    app.kubernetes.io/component: backend
    app.kubernetes.io/instance: daytrader10-wlo-instanton
    app.kubernetes.io/managed-by: websphere-liberty-operator
    app.kubernetes.io/name: daytrader10-wlo-instanton
    app.kubernetes.io/part-of: daytrader10-wlo-instanton
    liberty.websphere.ibm.com/name: daytrader10-wlo-instanton
  name: daytrader10-wlo-instanton
  namespace: nest-daytrader10-instanton
  ownerReferences:
  - apiVersion: liberty.websphere.ibm.com/v1
    blockOwnerDeletion: true
    controller: true
    kind: WebSphereLibertyApplication
    name: daytrader10-wlo-instanton
    uid: 4652f36d-91f4-4813-88a2-1cee731f74cc
  resourceVersion: "3166870"
  uid: baf5fedf-6792-4db1-bc36-fd3ad6ec8547
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: daytrader10-wlo-instanton
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "5"
        libertyOperator: WebSphere Liberty
        productChargedContainers: app
        productID: e7daacc46bbe4e2dacd2af49145a4723
        productMetric: PROCESSOR_VALUE_UNIT
        productName: IBM WebSphere Application Server
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: backend
        app.kubernetes.io/instance: daytrader10-wlo-instanton
        app.kubernetes.io/managed-by: websphere-liberty-operator
        app.kubernetes.io/name: daytrader10-wlo-instanton
        app.kubernetes.io/part-of: daytrader10-wlo-instanton
        liberty.websphere.ibm.com/name: daytrader10-wlo-instanton
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: daytrader10-wlo-instanton
              topologyKey: topology.kubernetes.io/zone
            weight: 50
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: daytrader10-wlo-instanton
              topologyKey: kubernetes.io/hostname
            weight: 50
      automountServiceAccountToken: true
      containers:
      - env:
        - name: httpPort
          value: "9080"
        - name: httpsPort
          value: "9443"
        - name: WLP_LOGGING_CONSOLE_FORMAT
          value: simple
        - name: dbHost
          value: app-db2
        - name: dbName
          value: TRADEDB
        - name: dbPort
          value: "50000"
        - name: dbUser
          value: db2inst1
        - name: dbPasswd
          valueFrom:
            secretKeyRef:
              key: dbpw
              name: db-credential
        - name: QueueManager
          value: QM1
        - name: mqHost
          value: app-mq
        - name: mqPasswd
          valueFrom:
            secretKeyRef:
              key: mqpw
              name: mqapp-credential
        - name: TLS_DIR
          value: /etc/x509/certs
        - name: SA_RESOURCE_VERSION
          value: "3166830"
        - name: WLP_LOGGING_CONSOLE_LOGLEVEL
          value: info
        - name: WLP_LOGGING_CONSOLE_SOURCE
          value: message,accessLog,ffdc,audit
        - name: IBM_HEAPDUMPDIR
          value: /serviceability
        - name: IBM_COREDIR
          value: /serviceability
        - name: IBM_JAVACOREDIR
          value: /serviceability
        - name: SEC_IMPORT_K8S_CERTS
          value: "true"
        - name: SERVICE_CERT_SECRET_RESOURCE_VERSION
          value: "3166854"
        image: docker-na-public.artifactory.swg-devops.com/hyc-wassvt-team-image-registry-docker-local/instanton/24.0.0.10/daytrader10:ol-full-java21-s390x
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: 9080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: app
        ports:
        - containerPort: 9443
          name: 9443-tcp
          protocol: TCP
        readinessProbe:
          failureThreshold: 5
          httpGet:
            path: /daytrader
            port: 9080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 1Gi
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - CHECKPOINT_RESTORE
            - SETPCAP
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: false
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        startupProbe:
          failureThreshold: 10
          httpGet:
            path: /health
            port: 9080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/x509/certs
          name: svc-certificate
          readOnly: true
        - mountPath: /serviceability
          name: serviceability
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: daytrader10-wlo-instanton
      serviceAccountName: daytrader10-wlo-instanton
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/instance: daytrader10-wlo-instanton
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
      - labelSelector:
          matchLabels:
            app.kubernetes.io/instance: daytrader10-wlo-instanton
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
      volumes:
      - name: svc-certificate
        secret:
          defaultMode: 420
          secretName: daytrader10-wlo-instanton-svc-tls-ocp
      - name: serviceability
        persistentVolumeClaim:
          claimName: daytrader10-wlo-instanton-serviceability
status:
  conditions:
  - lastTransitionTime: "2024-09-30T20:03:55Z"
    lastUpdateTime: "2024-09-30T20:03:55Z"
    message: Created new replica set "daytrader10-wlo-instanton-545748f787"
    reason: NewReplicaSetCreated
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-09-30T20:03:55Z"
    lastUpdateTime: "2024-09-30T20:03:55Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2024-09-30T20:03:55Z"
    lastUpdateTime: "2024-09-30T20:03:55Z"
    message: 'pods "daytrader10-wlo-instanton-545748f787-" is forbidden: unable to
      validate against any security context constraint: [provider "anyuid": Forbidden:
      not usable by user or serviceaccount, provider restricted-v2: .containers[0].capabilities.add:
      Invalid value: "CHECKPOINT_RESTORE": capability may not be added, provider restricted-v2:
      .containers[0].capabilities.add: Invalid value: "SETPCAP": capability may not
      be added, provider restricted-v2: .containers[0].allowPrivilegeEscalation: Invalid
      value: true: Allowing privilege escalation for containers is not allowed, provider
      "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2":
      Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden:
      not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden:
      not usable by user or serviceaccount, pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/app]:
      Forbidden: seccomp may not be set, provider "machine-api-termination-handler":
      Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2":
      Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden:
      not usable by user or serviceaccount, provider "hostaccess": Forbidden: not
      usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable
      by user or serviceaccount, provider "privileged": Forbidden: not usable by user
      or serviceaccount]'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure
  observedGeneration: 1
  unavailableReplicas: 1
leochr commented 1 month ago

@abdulmateen-1 Thank you for the resource details. Deployment is using the default service account that WLO created. It's not using the custom service accountinstanton-sa.

Did you configure service account in WebSphereLibertyApplication using .spec.serviceAccount.name field?

FYI, more details on service account configuration with WLO: https://www.ibm.com/docs/en/was-liberty/base?topic=operator-configuration-examples#cfg-r-commoncomp__cc-serviceacct

abdulmateen-1 commented 1 month ago

I tried the two ways to configure the service account. The first try is described above using .spec.serviceAccountName. When I tried .spec.serviceAccount.name the service account gets configured to instanton-sa but the pods never deployed.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"
    deployment.kubernetes.io/revision: "2"
  creationTimestamp: "2024-09-30T20:03:55Z"
  generation: 2
  labels:
    app.kubernetes.io/component: backend
    app.kubernetes.io/instance: daytrader10-wlo-instanton
    app.kubernetes.io/managed-by: websphere-liberty-operator
    app.kubernetes.io/name: daytrader10-wlo-instanton
    app.kubernetes.io/part-of: daytrader10-wlo-instanton
    liberty.websphere.ibm.com/name: daytrader10-wlo-instanton
  name: daytrader10-wlo-instanton
  namespace: nest-daytrader10-instanton
  ownerReferences:
  - apiVersion: liberty.websphere.ibm.com/v1
    blockOwnerDeletion: true
    controller: true
    kind: WebSphereLibertyApplication
    name: daytrader10-wlo-instanton
    uid: 4652f36d-91f4-4813-88a2-1cee731f74cc
  resourceVersion: "3873847"
  uid: baf5fedf-6792-4db1-bc36-fd3ad6ec8547
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: daytrader10-wlo-instanton
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "5"
        libertyOperator: WebSphere Liberty
        productChargedContainers: app
        productID: e7daacc46bbe4e2dacd2af49145a4723
        productMetric: PROCESSOR_VALUE_UNIT
        productName: IBM WebSphere Application Server
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: backend
        app.kubernetes.io/instance: daytrader10-wlo-instanton
        app.kubernetes.io/managed-by: websphere-liberty-operator
        app.kubernetes.io/name: daytrader10-wlo-instanton
        app.kubernetes.io/part-of: daytrader10-wlo-instanton
        liberty.websphere.ibm.com/name: daytrader10-wlo-instanton
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: daytrader10-wlo-instanton
              topologyKey: topology.kubernetes.io/zone
            weight: 50
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: daytrader10-wlo-instanton
              topologyKey: kubernetes.io/hostname
            weight: 50
      automountServiceAccountToken: true
      containers:
      - env:
        - name: httpPort
          value: "9080"
        - name: httpsPort
          value: "9443"
        - name: WLP_LOGGING_CONSOLE_FORMAT
          value: simple
        - name: dbHost
          value: app-db2
        - name: dbName
          value: TRADEDB
        - name: dbPort
          value: "50000"
        - name: dbUser
          value: db2inst1
        - name: dbPasswd
          valueFrom:
            secretKeyRef:
              key: dbpw
              name: db-credential
        - name: QueueManager
          value: QM1
        - name: mqHost
          value: app-mq
        - name: mqPasswd
          valueFrom:
            secretKeyRef:
              key: mqpw
              name: mqapp-credential
        - name: TLS_DIR
          value: /etc/x509/certs
        - name: SA_RESOURCE_VERSION
          value: "3035767"
        - name: WLP_LOGGING_CONSOLE_LOGLEVEL
          value: info
        - name: WLP_LOGGING_CONSOLE_SOURCE
          value: message,accessLog,ffdc,audit
        - name: IBM_HEAPDUMPDIR
          value: /serviceability
        - name: IBM_COREDIR
          value: /serviceability
        - name: IBM_JAVACOREDIR
          value: /serviceability
        - name: SEC_IMPORT_K8S_CERTS
          value: "true"
        - name: SERVICE_CERT_SECRET_RESOURCE_VERSION
          value: "3166854"
        image: docker-na-public.artifactory.swg-devops.com/hyc-wassvt-team-image-registry-docker-local/instanton/24.0.0.10/daytrader10:ol-full-java21-s390x
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: 9080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: app
        ports:
        - containerPort: 9443
          name: 9443-tcp
          protocol: TCP
        readinessProbe:
          failureThreshold: 5
          httpGet:
            path: /daytrader
            port: 9080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 1Gi
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - CHECKPOINT_RESTORE
            - SETPCAP
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: false
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        startupProbe:
          failureThreshold: 10
          httpGet:
            path: /health
            port: 9080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/x509/certs
          name: svc-certificate
          readOnly: true
        - mountPath: /serviceability
          name: serviceability
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: instanton-sa
      serviceAccountName: instanton-sa
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/instance: daytrader10-wlo-instanton
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
      - labelSelector:
          matchLabels:
            app.kubernetes.io/instance: daytrader10-wlo-instanton
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
      volumes:
      - name: svc-certificate
        secret:
          defaultMode: 420
          secretName: daytrader10-wlo-instanton-svc-tls-ocp
      - name: serviceability
        persistentVolumeClaim:
          claimName: daytrader10-wlo-instanton-serviceability
status:
  conditions:
  - lastTransitionTime: "2024-09-30T20:03:55Z"
    lastUpdateTime: "2024-09-30T20:03:55Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2024-09-30T20:03:55Z"
    lastUpdateTime: "2024-09-30T20:03:55Z"
    message: 'pods "daytrader10-wlo-instanton-545748f787-" is forbidden: unable to
      validate against any security context constraint: [provider "anyuid": Forbidden:
      not usable by user or serviceaccount, provider restricted-v2: .containers[0].capabilities.add:
      Invalid value: "CHECKPOINT_RESTORE": capability may not be added, provider restricted-v2:
      .containers[0].capabilities.add: Invalid value: "SETPCAP": capability may not
      be added, provider restricted-v2: .containers[0].allowPrivilegeEscalation: Invalid
      value: true: Allowing privilege escalation for containers is not allowed, provider
      "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2":
      Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden:
      not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden:
      not usable by user or serviceaccount, pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/app]:
      Forbidden: seccomp may not be set, provider "machine-api-termination-handler":
      Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2":
      Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden:
      not usable by user or serviceaccount, provider "hostaccess": Forbidden: not
      usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable
      by user or serviceaccount, provider "privileged": Forbidden: not usable by user
      or serviceaccount]'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure
  - lastTransitionTime: "2024-10-01T15:19:32Z"
    lastUpdateTime: "2024-10-01T15:19:32Z"
    message: Created new replica set "daytrader10-wlo-instanton-6495947dff"
    reason: NewReplicaSetCreated
    status: "True"
    type: Progressing
  observedGeneration: 2
  unavailableReplicas: 2
leochr commented 1 month ago

@abdulmateen-1 Thanks for providing the service account details in the Deployment. Could you please also provide the WebSphereLibertyApplication yaml ((including status) for this particular application?

leochr commented 1 month ago

@halim-lee Could you please investigate? I wonder if this is somehow related to the recent seccomp change to set RuntimeDefault. That didn't cause any errors with the restricted-v2 SCC though. The custom SCC cap-cr-scc created above for this app includes privileged permissions, but it'll be good to compare it to restricted-v2 to see if there is a difference in terms of seccomp. Another cause could be some difference in cluster configuration (Fyre vs zLinux).

abdulmateen-1 commented 1 month ago

This is the WebSphereLibertyApplication yaml file for this application

apiVersion: v1
items:
- apiVersion: liberty.websphere.ibm.com/v1
  kind: WebSphereLibertyApplication
  metadata:
    annotations:
      argocd.argoproj.io/sync-wave: "5"
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"liberty.websphere.ibm.com/v1","kind":"WebSphereLibertyApplication","metadata":{"annotations":{"argocd.argoproj.io/sync-wave":"5"},"name":"daytrader10-wlo-instanton","namespace":"nest-daytrader10-instanton"},"spec":{"applicationImage":"docker-na-public.artifactory.swg-devops.com/hyc-wassvt-team-image-registry-docker-local/instanton/24.0.0.10/daytrader10:ol-full-java21-s390x","autoscaling":{"maxReplicas":20,"minReplicas":1,"targetCPUUtilizationPercentage":50},"env":[{"name":"httpPort","value":"9080"},{"name":"httpsPort","value":"9443"},{"name":"WLP_LOGGING_CONSOLE_FORMAT","value":"simple"},{"name":"dbHost","value":"app-db2"},{"name":"dbName","value":"TRADEDB"},{"name":"dbPort","value":"50000"},{"name":"dbUser","value":"db2inst1"},{"name":"dbPasswd","valueFrom":{"secretKeyRef":{"key":"dbpw","name":"db-credential"}}},{"name":"QueueManager","value":"QM1"},{"name":"mqHost","value":"app-mq"},{"name":"mqPasswd","valueFrom":{"secretKeyRef":{"key":"mqpw","name":"mqapp-credential"}}}],"expose":true,"license":{"accept":true},"probes":{"liveness":{"failureThreshold":5,"httpGet":{"path":"/health","port":9080},"periodSeconds":10},"readiness":{"failureThreshold":5,"httpGet":{"path":"/daytrader","port":9080},"periodSeconds":10},"startup":{"failureThreshold":10,"httpGet":{"path":"/health","port":9080},"periodSeconds":10}},"pullPolicy":"Always","resources":{"limits":{"cpu":2,"memory":"2Gi"},"requests":{"cpu":"500m","memory":"1Gi"}},"securityContext":{"allowPrivilegeEscalation":true,"capabilities":{"add":["CHECKPOINT_RESTORE","SETPCAP"],"drop":["ALL"]},"privileged":false,"runAsNonRoot":true},"serviceAccount":{"mountToken":true,"name":"instanton-sa"},"serviceability":{"size":"1Gi"}}}
    creationTimestamp: "2024-09-30T20:03:54Z"
    finalizers:
    - finalizer.webspherelibertyapps.liberty.websphere.ibm.com
    generation: 7
    name: daytrader10-wlo-instanton
    namespace: nest-daytrader10-instanton
    resourceVersion: "3874687"
    uid: 4652f36d-91f4-4813-88a2-1cee731f74cc
  spec:
    applicationImage: docker-na-public.artifactory.swg-devops.com/hyc-wassvt-team-image-registry-docker-local/instanton/24.0.0.10/daytrader10:ol-full-java21-s390x
    applicationName: daytrader10-wlo-instanton
    autoscaling:
      maxReplicas: 20
      minReplicas: 1
      targetCPUUtilizationPercentage: 50
    env:
    - name: httpPort
      value: "9080"
    - name: httpsPort
      value: "9443"
    - name: WLP_LOGGING_CONSOLE_FORMAT
      value: simple
    - name: dbHost
      value: app-db2
    - name: dbName
      value: TRADEDB
    - name: dbPort
      value: "50000"
    - name: dbUser
      value: db2inst1
    - name: dbPasswd
      valueFrom:
        secretKeyRef:
          key: dbpw
          name: db-credential
    - name: QueueManager
      value: QM1
    - name: mqHost
      value: app-mq
    - name: mqPasswd
      valueFrom:
        secretKeyRef:
          key: mqpw
          name: mqapp-credential
    expose: true
    license:
      accept: true
      edition: IBM WebSphere Application Server
      productEntitlementSource: Standalone
    probes:
      liveness:
        failureThreshold: 5
        httpGet:
          path: /health
          port: 9080
        periodSeconds: 10
      readiness:
        failureThreshold: 5
        httpGet:
          path: /daytrader
          port: 9080
        periodSeconds: 10
      startup:
        failureThreshold: 10
        httpGet:
          path: /health
          port: 9080
        periodSeconds: 10
    pullPolicy: Always
    resources:
      limits:
        cpu: "2"
        memory: 2Gi
      requests:
        cpu: 500m
        memory: 1Gi
    securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        add:
        - CHECKPOINT_RESTORE
        - SETPCAP
        drop:
        - ALL
      privileged: false
      runAsNonRoot: true
    service:
      port: 9443
      type: ClusterIP
    serviceAccount:
      mountToken: true
      name: instanton-sa
    serviceability:
      size: 1Gi
  status:
    conditions:
    - lastTransitionTime: "2024-09-30T20:03:55Z"
      status: "True"
      type: Reconciled
    - lastTransitionTime: "2024-09-30T20:03:55Z"
      message: Resources are not ready.
      reason: ResourcesNotReady
      status: "False"
      type: Ready
    - lastTransitionTime: "2024-09-30T20:03:55Z"
      message: 'Deployment replicas ready: 0 < minReplicas: 1'
      reason: MinimumReplicasUnavailable
      status: "False"
      type: ResourcesReady
    endpoints:
    - name: Ingress
      scope: External
      type: Application
      uri: https://daytrader10-wlo-instanton-nest-daytrader10-instanton.apps.bphz4.cp.fyre.ibm.com
    imageReference: docker-na-public.artifactory.swg-devops.com/hyc-wassvt-team-image-registry-docker-local/instanton/24.0.0.10/daytrader10:ol-full-java21-s390x
    observedGeneration: 7
    references:
      saResourceVersion: "3035767"
      semeruGeneration: "1"
      svcCertSecretName: daytrader10-wlo-instanton-svc-tls-ocp
    versions:
      reconciled: 1.4.0
kind: List
metadata:
  resourceVersion: ""
tam512 commented 1 month ago

@tjwatson saw similar errors when deploying an InstantOn checkpoint image to x86 OCP 4.17.0 using OLO 1.4.0

SSguide-securityNamespaceNSsample-security
Oct 4, 2024, 2:57 PM
Generated from statefulset-controller
17 times in the last 6 minutes
create Pod guide-security-0 in StatefulSet guide-security failed error: pods "guide-security-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "CHECKPOINT_RESTORE": capability may not be added, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "SETPCAP": capability may not be added, provider restricted-v2: .containers[0].allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/app]: Forbidden: seccomp may not be set, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
halim-lee commented 1 month ago

From OCP 4.11, the default restricted-v2 SCC sets seccompProfile to runtime/default by default and throws a warning if seccompProfile is not set.

W0117 15:08:56.801191 1 warnings.go:67] would violate PodSecurity "restricted:v1.24": seccompProfile (pod or container "app" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

We delivered setting RuntimeDefault as the default value for seccompProfile for 1.4.0 release.

Now one of below is an available option:

tjwatson commented 1 month ago

I tried to add the seccompProfile like above, but now I am getting this error:

Generated from open-liberty-operator
2 times in the last 5 minutes
Secret "guide-security-svc-tls-ocp" was not found in namespace "sample-security", Secret "guide-security-svc-tls-ocp" not found

Here is my deploy yaml:

kind: OpenLibertyApplication
apiVersion: apps.openliberty.io/v1
metadata:
  name: guide-security
  labels:
    app: guide-security
spec:
  applicationImage: 'docker-na-public.artifactory.swg-devops.com/hyc-wassvt-team-image-registry-docker-local/tjwatson/guide-security-inston'
  expose: true
  replicas: 1
  serviceAccount:
    mountToken: true
  statefulSet: {}
  pullPolicy: Always
  pullSecret: artifactory-secret
  env:
    - name: WLP_LOGGING_CONSOLE_FORMAT
      value: simple
  serviceAccountName: liberty-instanton-scc
  securityContext:
    seccompProfile:
      type: Unconfined
    allowPrivilegeEscalation: true
    privileged: false
    runAsNonRoot: true
    capabilities:
      add:
      - CHECKPOINT_RESTORE
      - SETPCAP
      drop:
      - ALL

And here is the yaml used to create the scc:

kind: SecurityContextConstraints
metadata:
  annotations:
    kubernetes.io/description: criu-scc is based on the restricted SCC but removes any restrictions
      that prevent the restore image from running successfully.
  generation: 1
  name: scc-cap-cr
allowHostDirVolumePlugin: true
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
allowedCapabilities: null
apiVersion: security.openshift.io/v1
defaultAddCapabilities:
- CHECKPOINT_RESTORE
- SETPCAP
fsGroup:
  type: RunAsAny
groups:
- system:authenticated
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities:
- KILL
- MKNOD
- SETUID
- SETGID
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
supplementalGroups:
  type: RunAsAny
users: []
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- projected
- secret

Where I added the scc and named scc-cap-cr to the service account named liberty-instanton-ssc with the following oc commands:

  oc apply -f scc-cap-cr.yaml 
  oc create serviceaccount liberty-instanton-scc
  oc adm policy add-scc-to-user scc-cap-cr -z liberty-instanton-scc
leochr commented 1 month ago

@tjwatson Security best practices recommend using the runtime/default profile. Liberty Operator already sets securityContext to RuntimeDefault seccomp profile, so if it can be used, then remove the following from your config:

  securityContext:
    seccompProfile:
      type: Unconfined

If seccomp must be disabled for InstantOn, then unconfined must also be specified in the allowed seccomp profiles within the custom SCC as follows: Wildcard (*) could also be used instead but it's not recommended.

seccompProfiles:
  - unconfined
tam512 commented 1 month ago

Should we convert this defect into doc issue to have this documented in the Operator docs?

leochr commented 1 month ago

@tam512 Yes. We will work with ID to have this information documented.

@tjwatson let us know if the above suggestion worked. Thank you

tjwatson commented 1 month ago

No I still get 

create Pod guide-security-0 in StatefulSet guide-security failed error: pods "guide-security-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "CHECKPOINT_RESTORE": capability may not be added, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "SETPCAP": capability may not be added, provider restricted-v2: .containers[0].allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/app]: Forbidden: seccomp may not be set, provider scc-cap-cr-nop: .containers[0].capabilities.add: Invalid value: "SETPCAP": capability may not be added, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
[View all events](https://console-openshift-console.apps.tjwatson-instanton.cp.fyre.ibm.com/k8s/all-namespaces/events)
tam512 commented 1 month ago

@tjwatson, @Jonathan-Maciel just let me know about another change in operator that .spec.serviceAccountName is now deprecated and we should be using .spec.serviceAccount.name instead. It is documented here

.spec.serviceAccountName is now deprecated. The operator still looks up the value of .spec.serviceAccountName, but you must switch to using .spec.serviceAccount.name.

Try to update your app deploy yaml to have the following to see if it helps resolve the problem

spec:
  serviceAccount:
    mountToken: true
    name: liberty-instanton-scc
leochr commented 2 weeks ago

@tjwatson Are you still encountering the error/problem after configuring the service account, SCC and security context as per the above suggestions?