No terminations scheduled problem

[shreya8887]

Hello, I am facing similar issue which you already close it before #80. However, I am working on it from past few days and I am not able to solve it.

    I0721 19:41:07.985992       1 config.go:82] Successfully validated configs
    I0721 19:41:07.986015       1 main.go:54] Starting kube-monkey with v logging level 5 and local log directory /var/log/kube-monkey
    I0721 19:41:08.011412       1 kubemonkey.go:19] Debug mode detected!
    I0721 19:41:08.011452       1 kubemonkey.go:20] Status Update: Generating next schedule in 300 sec
    I0721 19:46:08.011598       1 schedule.go:64] Status Update: Generating schedule for terminations
    I0721 19:46:08.179514       1 schedule.go:57] Status Update: 0 terminations scheduled today
        ********** Today's schedule **********
    No terminations scheduled
        ********** End of schedule **********
    I0721 19:46:08.179589       1 kubemonkey.go:62] Status Update: Waiting to run scheduled terminations.
    I0721 19:46:08.179603       1 kubemonkey.go:76] Status Update: All terminations done.

Below are my config file for more details:

1. configmap.yaml

       ---
         apiVersion: v1
         kind: ConfigMap
         metadata:
           name: kube-monkey-config-map
           namespace: kube-system
         data:
           config.toml: |
             [kubemonkey]
             run_hour = 8
             start_hour = 10
             end_hour = 16
             grace period_sec=120
             blacklisted_namespaces = ["kube-system"]
             whitelisted_namespaces=["default"]
             time_zone="Canada/Eastern"
             [debug]
             enabled = true
             schedule_delay=300
             force_should_kill = true
             schedule_immediate_kill = true

2. deployment.yaml

       ---
         apiVersion: apps/v1
         kind: Deployment
         metadata:
           name: kube-monkey
           namespace: kube-system
         spec:
           replicas: 1
           selector:
             matchLabels:
               app: kube-monkey
           template:
             metadata:
               labels:
                 app: kube-monkey
             spec:
               serviceAccount:default
               serviceAccountName:default
               containers:
                 -  name: kube-monkey
                    command:
                      - "/kube-monkey"
                    args: ["-v=5", "-log_dir=/var/log/kube-monkey"]
                    image: ayushsobti/kube-monkey:v0.3.0
                    env:
                      - name: KUBE_MONKEY_ID
                        value: CLUSTER_A
                    volumeMounts:
                      - name: config-volume
                        mountPath: "/etc/kube-monkey"
               volumes:
                 - name: config-volume
                   configMap:
                     name: kube-monkey-config-map

3. docker-registry yaml(Victim application yaml)

         apiVersion: apps.openshift.io/v1
         kind: DeploymentConfig
         metadata:
           creationTimestamp: '2021-07-14T18:09:09Z'
           generation: 6
           labels:
             docker-registry: default
             kube-monkey/enabled: enabled
             kube-monkey/identifier: docker-registry
             kube-monkey/kill-mode: fixed
             kube-monkey/kill-value: '1'
             kube-monkey/mtbf: '1'
           name: docker-registry
           namespace: default
           resourceVersion: '313891'
           selfLink: >-
             /apis/apps.openshift.io/v1/namespaces/default/deploymentconfigs/docker-registry
           uid: 9604c68c-e4ce-11eb-bd40-08002774d158
         spec:
           replicas: 3
           revisionHistoryLimit: 10
           selector:
             docker-registry: default
           strategy:
             activeDeadlineSeconds: 21600
             resources: {}
             rollingParams:
               intervalSeconds: 1
               maxSurge: 25%
               maxUnavailable: 25%
               timeoutSeconds: 600
               updatePeriodSeconds: 1
             type: Rolling
           template:
             metadata:
               creationTimestamp: null
               labels:
                 docker-registry: default
                 kube-monkey/enabled: enabled
                 kube-monkey/identifier: docker-registry
                 kube-monkey/kill-mode: fixed
                 kube-monkey/kill-value: '1'
                 kube-monkey/mtbf: '1'
             spec:
               containers:
                 - env:
                     - name: REGISTRY_HTTP_ADDR
                       value: ':5000'
                     - name: REGISTRY_HTTP_NET
                       value: tcp
                     - name: REGISTRY_HTTP_SECRET
                       value: t84he+iqXzoG4l3QNS4Y02m4TxV2P7Ww53k/BvoNZKU=
                     - name: REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA
                       value: 'false'
                   image: 'openshift/origin-docker-registry:v3.11.0'
                   imagePullPolicy: IfNotPresent
                   livenessProbe:
                     failureThreshold: 3
                     httpGet:
                       path: /healthz
                       port: 5000
                       scheme: HTTP
                     initialDelaySeconds: 10
                     periodSeconds: 10
                     successThreshold: 1
                     timeoutSeconds: 5
                   name: registry
                   ports:
                     - containerPort: 5000
                       protocol: TCP
                   readinessProbe:
                     failureThreshold: 3
                     httpGet:
                       path: /healthz
                       port: 5000
                       scheme: HTTP
                     periodSeconds: 10
                     successThreshold: 1
                     timeoutSeconds: 5
                   resources:
                     requests:
                       cpu: 100m
                       memory: 256Mi
                   securityContext:
                     privileged: true
                   terminationMessagePath: /dev/termination-log
                   terminationMessagePolicy: File
                   volumeMounts:
                     - mountPath: /registry
                       name: registry-storage
               dnsPolicy: ClusterFirst
               restartPolicy: Always
               schedulerName: default-scheduler
               securityContext: {}
               serviceAccount: registry
               serviceAccountName: registry
               terminationGracePeriodSeconds: 30
               volumes:
                 - hostPath:
                     path: /var/lib/minishift/base/openshift.local.pv/registry
                     type: ''
                   name: registry-storage
           test: false
           triggers:
             - type: ConfigChange
         status:
           availableReplicas: 3
           conditions:
             - lastTransitionTime: '2021-07-20T20:07:16Z'
               lastUpdateTime: '2021-07-20T20:07:16Z'
               message: Deployment config has minimum availability.
               status: 'True'
               type: Available
             - lastTransitionTime: '2021-07-20T21:01:15Z'
               lastUpdateTime: '2021-07-20T21:01:18Z'
               message: replication controller "docker-registry-4" successfully rolled out
               reason: NewReplicationControllerAvailable
               status: 'True'
               type: Progressing
           details:
             causes:
               - type: ConfigChange
             message: config change
           latestVersion: 4
           observedGeneration: 6
           readyReplicas: 3
           replicas: 3
           unavailableReplicas: 0
           updatedReplicas: 3

I have also tried with same configmap.yaml that Kube-monkey repository has but hit the wall.

I have deploy Kube-monkey on OCP 4. It is using Kubernetes Version 1.19.0. Note: I have also tried same thing with OpenShift 3.11.

Could anyone please help me with this problem ?

[worldtiki]

Hi @shreya8887! I'm not familiar with openshift so I'm not sure if I'll be able to help much.

But you can see here that kube-monkey only targets apps/v1 Deployments but in your case you are using a apps.openshift.io/v1 DeploymentConfig.

Does that create a regular deployment behind the scenes? And if so, does it carry over the labels?

[worldtiki]

It looks like someone tried to solve this before and that it requires code changes: https://github.com/faust64/kube-monkey/commit/6b9956dc6157b05387c0b1e5c569aa95e6903310