Placeholder for nodeSelectors, Tolerations and resources:{} needs to be enabled in the final workflow for experiment

[biplabMazumdar]

What is the issue? While creating the experiment , there is no way to set the spec params like nodeSelectors, Tolerations and resources:{}.

Impact of the above issue The experiment pods fails to launch due to the above constraint.

Workaround I tried modying the final workflow to pass on the spec params , but thats not working. I tried podSpecPatch , that also did not work Currently no workaround I could find.

Errors / Events 49m Normal WorkflowRunning workflow/exp2-1709391345601 Workflow Running 49m Warning WorkflowFailed workflow/exp2-1709391345601 step group deemed errored due to child exp2-1709391345601[0].install-chaos-faults error: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3containerlimits-bf1dfd0b98bb4e52b775] container has no resource limits... 49m Normal WorkflowNodeRunning workflow/exp2-1709391345601 Running node exp2-1709391345601: step group deemed errored due to child exp2-1709391345601[0].install-chaos-faults error: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3containerlimits-bf1dfd0b98bb4e52b775] container has no resource limits... 49m Warning WorkflowNodeFailed workflow/exp2-1709391345601 Failed node exp2-1709391345601: step group deemed errored due to child exp2-1709391345601[0].install-chaos-faults error: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3containerlimits-bf1dfd0b98bb4e52b775] container has no resource limits... 49m Normal WorkflowNodeRunning workflow/exp2-1709391345601 Running node exp2-1709391345601[0]: step group deemed errored due to child exp2-1709391345601[0].install-chaos-faults error: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3containerlimits-bf1dfd0b98bb4e52b775] container has no resource limits... 49m Warning WorkflowNodeError workflow/exp2-1709391345601 Error node exp2-1709391345601[0]: step group deemed errored due to child exp2-1709391345601[0].install-chaos-faults error: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3containerlimits-bf1dfd0b98bb4e52b775] container has no resource limits... 49m Normal WorkflowNodeRunning workflow/exp2-1709391345601 Running node exp2-1709391345601[0].install-chaos-faults: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3containerlimits-bf1dfd0b98bb4e52b775] container has no resource limits... 49m Warning WorkflowNodeError workflow/exp2-1709391345601 Error node exp2-1709391345601[0].install-chaos-faults: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3containerlimits-bf1dfd0b98bb4e52b775] container has no resource limits... 11m Warning WorkflowFailed workflow/exp2-1709393625581 invalid spec: templates.exp2.steps[0].install-chaos-faults templates.install-chaos-faults: failed to resolve {{inputs.parameters.effect1}} 3m48s Normal NotTriggerScaleUp pod/pod-cpu-hog-cwtdzo-t9j2n pod didn't trigger scale-up: 1 node(s) had untolerated taint {app: plmrs}, 1 max node group size reached 13m Normal NotTriggerScaleUp pod/pod-cpu-hog-cwtdzo-t9j2n pod didn't trigger scale-up: 1 max node group size reached, 1 node(s) had untolerated taint {app: plmrs} 6m49s Warning FailedScheduling pod/pod-cpu-hog-cwtdzo-t9j2n 0/12 nodes are available: 3 node(s) had untolerated taint {CriticalAddonsOnly: true}, 9 node(s) had untolerated taint {app: plmrs}. preemption: 0/12 nodes are available: 12 Preemption is not helpful for scheduling..

Current Modified Workflow File

kind: Workflow apiVersion: argoproj.io/v1alpha1 metadata: name: exp2 namespace: plm labels: infra_id: b50e8dc3-9f0a-4e15-bf0f-4061c3eb5c44 revision_id: f9bf2725-6bbb-43f6-b674-c4171df5b220 workflow_id: e0717cca-59f0-4efe-98a1-36b2106733b5 workflows.argoproj.io/controller-instanceid: b50e8dc3-9f0a-4e15-bf0f-4061c3eb5c44 spec: templates:

• name: exp2 inputs: {} outputs: {} metadata: {} steps:
  • • name: install-chaos-faults template: install-chaos-faults arguments: parameters:
      • name: effect1 value: NoSchedule
      • name: key1 value: app
      • name: operator1 value: Equal
      • name: value1 value: plmrs
      • name: effect2 value: NoExecute
      • name: key2 value: node.kubernetes.io/not-ready
      • name: operator2 value: Exists
      • name: tolerationSeconds1 value: "300"
      • name: effect3 value: NoExecute
      • name: key3 value: node.kubernetes.io/unreachable
      • name: operator3 value: Exists
      • name: tolerationSeconds2 value: "300"
      • name: effect4 value: NoSchedule
      • name: key4 value: node.kubernetes.io/memory-pressure
      • name: operator4 value: Exists
  • • name: pod-cpu-hog-a5o template: pod-cpu-hog-a5o arguments: parameters:
      • name: effect1 value: NoSchedule
      • name: key1 value: app
      • name: operator1 value: Equal
      • name: value1 value: plmrs
      • name: effect2 value: NoExecute
      • name: key2 value: node.kubernetes.io/not-ready
      • name: operator2 value: Exists
      • name: tolerationSeconds1 value: "300"
      • name: effect3 value: NoExecute
      • name: key3 value: node.kubernetes.io/unreachable
      • name: operator3 value: Exists
      • name: tolerationSeconds2 value: "300"
      • name: effect4 value: NoSchedule
      • name: key4 value: node.kubernetes.io/memory-pressure
      • name: operator4 value: Exists
  • • name: cleanup-chaos-resources template: cleanup-chaos-resources arguments: parameters:
      • name: effect1 value: NoSchedule
      • name: key1 value: app
      • name: operator1 value: Equal
      • name: value1 value: plmrs
      • name: effect2 value: NoExecute
      • name: key2 value: node.kubernetes.io/not-ready
      • name: operator2 value: Exists
      • name: tolerationSeconds1 value: "300"
      • name: effect3 value: NoExecute
      • name: key3 value: node.kubernetes.io/unreachable
      • name: operator3 value: Exists
      • name: tolerationSeconds2 value: "300"
      • name: effect4 value: NoSchedule
      • name: key4 value: node.kubernetes.io/memory-pressure
      • name: operator4 value: Exists

• name: install-chaos-faults inputs: artifacts:

  • name: pod-cpu-hog-a5o path: /tmp/pod-cpu-hog-a5o.yaml raw: data: > apiVersion: litmuschaos.io/v1alpha1

        description:
          message: |
            Injects CPU consumption on pods belonging to an app deployment
        kind: ChaosExperiment
    
        metadata:
          name: pod-cpu-hog
          labels:
            name: pod-cpu-hog
            app.kubernetes.io/part-of: litmus
            app.kubernetes.io/component: chaosexperiment
            app.kubernetes.io/version: 3.1.0
        spec:
          definition:
            scope: Namespaced
            permissions:
              - apiGroups:
                  - ""
                resources:
                  - pods
                verbs:
                  - create
                  - delete
                  - get
                  - list
                  - patch
                  - update
                  - deletecollection
              - apiGroups:
                  - ""
                resources:
                  - events
                verbs:
                  - create
                  - get
                  - list
                  - patch
                  - update
              - apiGroups:
                  - ""
                resources:
                  - configmaps
                verbs:
                  - get
                  - list
              - apiGroups:
                  - ""
                resources:
                  - pods/log
                verbs:
                  - get
                  - list
                  - watch
              - apiGroups:
                  - ""
                resources:
                  - pods/exec
                verbs:
                  - get
                  - list
                  - create
              - apiGroups:
                  - apps
                resources:
                  - deployments
                  - statefulsets
                  - replicasets
                  - daemonsets
                verbs:
                  - list
                  - get
              - apiGroups:
                  - apps.openshift.io
                resources:
                  - deploymentconfigs
                verbs:
                  - list
                  - get
              - apiGroups:
                  - ""
                resources:
                  - replicationcontrollers
                verbs:
                  - get
                  - list
              - apiGroups:
                  - argoproj.io
                resources:
                  - rollouts
                verbs:
                  - list
                  - get
              - apiGroups:
                  - batch
                resources:
                  - jobs
                verbs:
                  - create
                  - list
                  - get
                  - delete
                  - deletecollection
              - apiGroups:
                  - litmuschaos.io
                resources:
                  - chaosengines
                  - chaosexperiments
                  - chaosresults
                verbs:
                  - create
                  - list
                  - get
                  - patch
                  - update
                  - delete
            image: litmuschaos/go-runner:3.1.0
            imagePullPolicy: Always
            args:
              - -c
              - ./experiments -name pod-cpu-hog
            command:
              - /bin/bash
            env:
              - name: TOTAL_CHAOS_DURATION
                value: "60"
              - name: CPU_CORES
                value: "1"
              - name: CPU_LOAD
                value: "100"
              - name: PODS_AFFECTED_PERC
                value: ""
              - name: RAMP_TIME
                value: ""
              - name: LIB_IMAGE
                value: litmuschaos/go-runner:3.1.0
              - name: STRESS_IMAGE
                value: alexeiled/stress-ng:latest-ubuntu
              - name: CONTAINER_RUNTIME
                value: containerd
              - name: SOCKET_PATH
                value: /run/containerd/containerd.sock
              - name: TARGET_CONTAINER
                value: ""
              - name: TARGET_PODS
                value: ""
              - name: DEFAULT_HEALTH_CHECK
                value: "false"
              - name: NODE_LABEL
                value: ""
              - name: SEQUENCE
                value: parallel
            labels:
              name: pod-cpu-hog
              app.kubernetes.io/part-of: litmus
              app.kubernetes.io/component: experiment-job
              app.kubernetes.io/runtime-api-usage: "true"
              app.kubernetes.io/version: 3.1.0
          nodeSelector:
            app: plmrs-apps
          tolerations:
            - key: app
              operator: Equal
              value: plmrs
              effect: NoSchedule
            - key: node.kubernetes.io/not-ready
              operator: Exists
              effect: NoExecute
              tolerationSeconds: 300
            - key: node.kubernetes.io/unreachable
              operator: Exists
              effect: NoExecute
              tolerationSeconds: 300
            - key: node.kubernetes.io/memory-pressure
              operator: Exists  
              effect: NoSchedule

    outputs: {} metadata: {} container: name: "" image: litmuschaos/k8s:2.11.0 command:

  • sh
  • -c args:
  • kubectl apply -f /tmp/ -n {{workflow.parameters.adminModeNamespace}} && sleep 30 resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi podSpecPatch: '{"tolerations":[{"effect":"{{inputs.parameters.effect1}}","key":"{{inputs.parameters.key1}}","operator":"{{inputs.parameters.operator1}}","value":"{{inputs.parameters.value1}}"},{"effect":"{{inputs.parameters.effect2}}","key":"{{inputs.parameters.key2}}","operator":"{{inputs.parameters.operator2}}","tolerationSeconds":"{{inputs.parameters.tolerationSeconds1}}"},{"effect":"{{inputs.parameters.effect2}}","key":"{{inputs.parameters.key2}}","operator":"{{inputs.parameters.operator2}}","tolerationSeconds":"{{inputs.parameters.tolerationSeconds2}}"},{"effect":"{{inputs.parameters.effect4}}","key":"{{inputs.parameters.key4}}","operator":"{{inputs.parameters.operator4}}"}]}'
• name: cleanup-chaos-resources inputs: {} outputs: {} metadata: {} container: name: "" image: litmuschaos/k8s:2.11.0 command:
  • sh
  • -c args:
  • kubectl delete chaosengine -l workflow_run_id={{workflow.uid}} -n {{workflow.parameters.adminModeNamespace}} resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi podSpecPatch: '{"tolerations":[{"effect":"{{inputs.parameters.effect1}}","key":"{{inputs.parameters.key1}}","operator":"{{inputs.parameters.operator1}}","value":"{{inputs.parameters.value1}}"},{"effect":"{{inputs.parameters.effect2}}","key":"{{inputs.parameters.key2}}","operator":"{{inputs.parameters.operator2}}","tolerationSeconds":"{{inputs.parameters.tolerationSeconds1}}"},{"effect":"{{inputs.parameters.effect2}}","key":"{{inputs.parameters.key2}}","operator":"{{inputs.parameters.operator2}}","tolerationSeconds":"{{inputs.parameters.tolerationSeconds2}}"},{"effect":"{{inputs.parameters.effect4}}","key":"{{inputs.parameters.key4}}","operator":"{{inputs.parameters.operator4}}"}]}'
• name: pod-cpu-hog-a5o inputs: artifacts:
  • name: pod-cpu-hog-a5o path: /tmp/chaosengine-pod-cpu-hog-a5o.yaml raw: data: | apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: namespace: "{{workflow.parameters.adminModeNamespace}}" labels: workflow_run_id: "{{ workflow.uid }}" workflow_name: exp2 annotations: probeRef: '[{"name":"http-validator","mode":"SOT"}]' generateName: pod-cpu-hog-a5o spec: engineState: active appinfo: appns: plm applabel: app=rsproduct-validator appkind: deployment chaosServiceAccount: litmus-admin nodeSelector: app: plmrs-apps tolerations:
    • key: app operator: Equal value: plmrs effect: NoSchedule
    • key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 300
    • key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 300
    • key: node.kubernetes.io/memory-pressure operator: Exists effect: NoSchedule experiments:
    • name: pod-cpu-hog spec: components: env:
      • name: TOTAL_CHAOS_DURATION value: "60"
      • name: CPU_CORES value: "1"
      • name: CPU_LOAD value: "100"
      • name: PODS_AFFECTED_PERC value: ""
      • name: RAMP_TIME value: ""
      • name: LIB_IMAGE value: litmuschaos/go-runner:3.1.0
      • name: STRESS_IMAGE value: alexeiled/stress-ng:latest-ubuntu
      • name: CONTAINER_RUNTIME value: containerd
      • name: SOCKET_PATH value: /run/containerd/containerd.sock
      • name: TARGET_CONTAINER value: ""
      • name: TARGET_PODS value: ""
      • name: DEFAULT_HEALTH_CHECK value: "false"
      • name: NODE_LABEL value: ""
      • name: SEQUENCE value: parallel outputs: {} metadata: labels: weight: "10" container: name: "" image: docker.io/litmuschaos/litmus-checker:2.11.0 args:
  • -file=/tmp/chaosengine-pod-cpu-hog-a5o.yaml
  • -saveName=/tmp/engine-name resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi podSpecPatch: '{"tolerations":[{"effect":"{{inputs.parameters.effect1}}","key":"{{inputs.parameters.key1}}","operator":"{{inputs.parameters.operator1}}","value":"{{inputs.parameters.value1}}"},{"effect":"{{inputs.parameters.effect2}}","key":"{{inputs.parameters.key2}}","operator":"{{inputs.parameters.operator2}}","tolerationSeconds":"{{inputs.parameters.tolerationSeconds1}}"},{"effect":"{{inputs.parameters.effect2}}","key":"{{inputs.parameters.key2}}","operator":"{{inputs.parameters.operator2}}","tolerationSeconds":"{{inputs.parameters.tolerationSeconds2}}"},{"effect":"{{inputs.parameters.effect4}}","key":"{{inputs.parameters.key4}}","operator":"{{inputs.parameters.operator4}}"}]}' entrypoint: exp2 arguments: parameters:
    • name: adminModeNamespace value: plm serviceAccountName: argo-chaos nodeSelector: app: plmrs-apps tolerations:
• key: app operator: Equal value: plmrs effect: NoSchedule
• key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 300
• key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 300
• key: node.kubernetes.io/memory-pressure operator: Exists effect: NoSchedule podGC: strategy: OnWorkflowCompletion securityContext: runAsUser: 1000 runAsNonRoot: true status: {}

[biplabMazumdar]

Is this feature already available ? Can someone please comment. I was not able to set the tolerations required and hence the required litmus pods and experiment jobs were not getting scheduled.