amitbhatt818 commented 4 years ago

What happened:

Openebs reinstallation job is failing in pipeline

Steps we are performing :

We are installing openebs using operator yaml

## Installing openebs on the cluster
- name: Installing openebs on the cluster
shell: kubectl apply -f https://openebs.github.io/charts/openebs-operator-1.7.0.yaml

After this we are checking for stats of all the components

    ## Checking the openebs Installation    
    - name: Fetch OpenEBS control plane pods state
      shell: kubectl get pods -n openebs  | grep {{ item }} | awk '{print $3}' | awk -F':' '{print $1}' | tail -n 1
      register: app_status
      until: app_status.stdout == 'Running'
      with_items:
        - "{{ openebs_components }}"
      retries: 30
      delay: 10

    ## Get the container Status of the openebs pods
    - name: Get the container status of the openebs pods.
      shell: >
        kubectl get pod -n openebs --no-headers
        -o jsonpath='{.items[*].status.containerStatuses[*].ready}' | tr ' ' '\n' | uniq
      args:
        executable: /bin/bash
      register: containerStatus
      until: "containerStatus.stdout == 'true'"
      retries: 30
      delay: 10

And at last we are trying to reinstall openbs using director.

    - name: Giving required variables for openebs installation
      uri:
        url: "{{ director_url }}/v3/groups/{{ group_id }}/clusters/{{ cluster_id }}/openebses"
        method: POST
        url_username: "{{ username.stdout }}"
        url_password: "{{ password.stdout }}"
        force_basic_auth: yes
        return_content: yes
        body_format: json
        body: '{ "clusterId": "{{ cluster_id }}","creatorId": "{{ group_id }}","projectId": "{{ project_id }}","templateId": "{{ template_id }}","namespace": "{{ namespace }}","defaultDirectory": "{{ default_directory }}","dockerRegistry": "{{ docker_registry }}","includeDeviceFilters": "{{ include_device_filters }}","excludeDeviceFilters": "{{ exclude_device_filters }}","cpuResourceLimit": "{{ cpu_resource_limit }}","memoryResourceLimit": " {{ memory_resource_limit }}","installationMode": "{{ installation_mode }}" }'         
        status_code: 405
      register: get_openebs

We are giving post request to openebses end point and checking for status: 405 but it is giving status code: 201 Logs:

TASK [Giving required variables for openebs installation] **********************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "connection": "close", "content": "{\"id\":\"1o5\",\"type\":\"openebs\",\"links\":{\"self\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\",\"cluster\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\\/cluster\",\"creator\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\\/creator\",\"openebsJobs\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\\/openebsjobs\",\"project\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\\/project\",\"template\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\\/template\",\"remove\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\",\"update\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\"},\"actions\":{\"openebsinstall\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\\/?action=openebsinstall\",\"selectopenebsnodes\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\\/?action=selectopenebsnodes\",\"upgradecontrolplane\":\"http:\\/\\/10.43.10.13\\/v3\\/groups\\/1a22\\/openebses\\/1o5\\/?action=upgradecontrolplane\"},\"baseType\":\"openebs\",\"state\":\"creating\",\"clusterId\":\"1c4\",\"cpuResourceLimit\":\"100m\",\"created\":\"2020-07-03T05:33:15Z\",\"createdTS\":1593754395000,\"creatorId\":\"1a21\",\"defaultDirectory\":\"\\/var\\/openebs\",\"dockerRegistry\":\"mayadataio\",\"excludeDeviceFilters\":\"loop,\\/dev\\/fd0,\\/dev\\/sr0,\\/dev\\/ram,\\/dev\\/dm-,\\/dev\\/md,\\/dev\\/rbd\",\"includeDeviceFilters\":\"\",\"installMayastor\":false,\"installationManifest\":{\"metadata\":{\"name\":\"install-openebs-1.11.0-ee-rc1\",\"namespace\":\"openebs\",\"labels\":{\"name\":\"openebs-upgrade\"}},\"apiVersion\":\"dao.mayadata.io\\/v1alpha1\",\"kind\":\"OpenEBS\",\"spec\":{\"version\":\"1.11.0-ee\",\"defaultStoragePath\":\"\\/var\\/openebs\",\"createDefaultStorageConfig\":true,\"imagePrefix\":\"mayadataio\",\"imageTagSuffix\":\"RC1\",\"imagePullPolicy\":\"IfNotPresent\",\"resources\":{\"requests\":{\"memory\":null,\"cpu\":null},\"limits\":{\"memory\":\"200Mi\",\"cpu\":\"100m\"}},\"apiServer\":{\"enabled\":true,\"imageTag\":null,\"replicas\":1,\"cstorSparsePool\":{\"enabled\":false},\"nodeSelector\":{\"mayadata.io\\/control-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}]},\"provisioner\":{\"enabled\":true,\"imageTag\":null,\"replicas\":1,\"nodeSelector\":{\"mayadata.io\\/control-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}]},\"localProvisioner\":{\"enabled\":true,\"imageTag\":null,\"replicas\":1,\"nodeSelector\":{\"mayadata.io\\/control-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}]},\"snapshotOperator\":{\"enabled\":true,\"controller\":{\"imageTag\":null},\"provisioner\":{\"imageTag\":null},\"replicas\":1,\"nodeSelector\":{\"mayadata.io\\/control-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}]},\"ndmDaemon\":{\"enabled\":true,\"imageTag\":null,\"replicas\":1,\"sparse\":{\"path\":null,\"size\":null,\"count\":null},\"filters\":{\"osDisk\":{\"enabled\":null,\"exclude\":\"\\/,\\/etc\\/hosts,\\/boot\"},\"vendor\":{\"enabled\":null,\"exclude\":\"CLOUDBYT,OpenEBS\"},\"path\":{\"enabled\":null,\"exclude\":\"loop,\\/dev\\/fd0,\\/dev\\/sr0,\\/dev\\/ram,\\/dev\\/dm-,\\/dev\\/md,\\/dev\\/rbd\",\"include\":null}},\"probes\":{\"udev\":{\"enabled\":true},\"smart\":{\"enabled\":true},\"seachest\":{\"enabled\":false}},\"nodeSelector\":{\"mayadata.io\\/data-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}]},\"ndmOperator\":{\"enabled\":true,\"imageTag\":null,\"replicas\":1,\"nodeSelector\":{\"mayadata.io\\/control-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}]},\"jivaConfig\":{\"imageTag\":null,\"replicas\":1},\"cstorConfig\":{\"pool\":{\"imageTag\":null},\"poolMgmt\":{\"imageTag\":null},\"target\":{\"imageTag\":null},\"volumeMgmt\":{\"imageTag\":null},\"csi\":{\"csiController\":{\"enabled\":null,\"imageTag\":null},\"csiNode\":{\"enabled\":null,\"imageTag\":null,\"iscsiPath\":null}},\"cspiMgmt\":{\"imageTag\":null},\"volumeManager\":{\"imageTag\":null},\"admissionServer\":{\"enabled\":null,\"imageTag\":null,\"replicas\":null,\"resources\":null,\"nodeSelector\":{\"mayadata.io\\/control-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}],\"affinity\":null},\"cspcOperator\":{\"enabled\":null,\"imageTag\":null,\"replicas\":null,\"resources\":null,\"nodeSelector\":{\"mayadata.io\\/control-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}],\"affinity\":null},\"cvcOperator\":{\"enabled\":null,\"imageTag\":null,\"replicas\":null,\"resources\":null,\"nodeSelector\":{\"mayadata.io\\/control-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}],\"affinity\":null}},\"admissionServer\":{\"enabled\":true,\"imageTag\":null,\"replicas\":1,\"nodeSelector\":{\"mayadata.io\\/control-plane\":\"true\"},\"tolerations\":[{\"operator\":\"Exists\"}],\"affinity\":null},\"mayastorConfig\":{\"moac\":{\"enabled\":false,\"imageTag\":null},\"mayastor\":{\"enabled\":false,\"mayastor\":{\"imageTag\":null},\"mayastorGrpc\":{\"imageTag\":null}}},\"helper\":{\"imageTag\":null},\"policies\":{\"monitoring\":{\"enabled\":true,\"imageTag\":null}},\"analytics\":{\"enabled\":true,\"pingInterval\":\"24h\"},\"options\":null}},\"installationMode\":\"basic\",\"kind\":\"openebs\",\"memoryResourceLimit\":\" \",\"namespace\":\"openebs\",\"projectId\":\"1p3\",\"removed\":null,\"templateId\":\"1ot2\",\"transitioning\":\"yes\",\"transitioningMessage\":\"Creating\",\"uuid\":\"6675d584-8375-462f-ad82-7a629094c396\",\"version\":null}", "content_type": "application/json; charset=utf-8", "cookies": {"PL": "DirectorOnline"}, "cookies_string": "PL=DirectorOnline", "date": "Fri, 03 Jul 2020 05:33:14 GMT", "elapsed": 0, "expires": "Thu, 01 Jan 1970 00:00:00 GMT", "json": {"actions": {"openebsinstall": "http://10.43.10.13/v3/groups/1a22/openebses/1o5/?action=openebsinstall", "selectopenebsnodes": "http://10.43.10.13/v3/groups/1a22/openebses/1o5/?action=selectopenebsnodes", "upgradecontrolplane": "http://10.43.10.13/v3/groups/1a22/openebses/1o5/?action=upgradecontrolplane"}, "baseType": "openebs", "clusterId": "1c4", "cpuResourceLimit": "100m", "created": "2020-07-03T05:33:15Z", "createdTS": 1593754395000, "creatorId": "1a21", "defaultDirectory": "/var/openebs", "dockerRegistry": "mayadataio", "excludeDeviceFilters": "loop,/dev/fd0,/dev/sr0,/dev/ram,/dev/dm-,/dev/md,/dev/rbd", "id": "1o5", "includeDeviceFilters": "", "installMayastor": false, "installationManifest": {"apiVersion": "dao.mayadata.io/v1alpha1", "kind": "OpenEBS", "metadata": {"labels": {"name": "openebs-upgrade"}, "name": "install-openebs-1.11.0-ee-rc1", "namespace": "openebs"}, "spec": {"admissionServer": {"affinity": null, "enabled": true, "imageTag": null, "nodeSelector": {"mayadata.io/control-plane": "true"}, "replicas": 1, "tolerations": [{"operator": "Exists"}]}, "analytics": {"enabled": true, "pingInterval": "24h"}, "apiServer": {"cstorSparsePool": {"enabled": false}, "enabled": true, "imageTag": null, "nodeSelector": {"mayadata.io/control-plane": "true"}, "replicas": 1, "tolerations": [{"operator": "Exists"}]}, "createDefaultStorageConfig": true, "cstorConfig": {"admissionServer": {"affinity": null, "enabled": null, "imageTag": null, "nodeSelector": {"mayadata.io/control-plane": "true"}, "replicas": null, "resources": null, "tolerations": [{"operator": "Exists"}]}, "csi": {"csiController": {"enabled": null, "imageTag": null}, "csiNode": {"enabled": null, "imageTag": null, "iscsiPath": null}}, "cspcOperator": {"affinity": null, "enabled": null, "imageTag": null, "nodeSelector": {"mayadata.io/control-plane": "true"}, "replicas": null, "resources": null, "tolerations": [{"operator": "Exists"}]}, "cspiMgmt": {"imageTag": null}, "cvcOperator": {"affinity": null, "enabled": null, "imageTag": null, "nodeSelector": {"mayadata.io/control-plane": "true"}, "replicas": null, "resources": null, "tolerations": [{"operator": "Exists"}]}, "pool": {"imageTag": null}, "poolMgmt": {"imageTag": null}, "target": {"imageTag": null}, "volumeManager": {"imageTag": null}, "volumeMgmt": {"imageTag": null}}, "defaultStoragePath": "/var/openebs", "helper": {"imageTag": null}, "imagePrefix": "mayadataio", "imagePullPolicy": "IfNotPresent", "imageTagSuffix": "RC1", "jivaConfig": {"imageTag": null, "replicas": 1}, "localProvisioner": {"enabled": true, "imageTag": null, "nodeSelector": {"mayadata.io/control-plane": "true"}, "replicas": 1, "tolerations": [{"operator": "Exists"}]}, "mayastorConfig": {"mayastor": {"enabled": false, "mayastor": {"imageTag": null}, "mayastorGrpc": {"imageTag": null}}, "moac": {"enabled": false, "imageTag": null}}, "ndmDaemon": {"enabled": true, "filters": {"osDisk": {"enabled": null, "exclude": "/,/etc/hosts,/boot"}, "path": {"enabled": null, "exclude": "loop,/dev/fd0,/dev/sr0,/dev/ram,/dev/dm-,/dev/md,/dev/rbd", "include": null}, "vendor": {"enabled": null, "exclude": "CLOUDBYT,OpenEBS"}}, "imageTag": null, "nodeSelector": {"mayadata.io/data-plane": "true"}, "probes": {"seachest": {"enabled": false}, "smart": {"enabled": true}, "udev": {"enabled": true}}, "replicas": 1, "sparse": {"count": null, "path": null, "size": null}, "tolerations": [{"operator": "Exists"}]}, "ndmOperator": {"enabled": true, "imageTag": null, "nodeSelector": {"mayadata.io/control-plane": "true"}, "replicas": 1, "tolerations": [{"operator": "Exists"}]}, "options": null, "policies": {"monitoring": {"enabled": true, "imageTag": null}}, "provisioner": {"enabled": true, "imageTag": null, "nodeSelector": {"mayadata.io/control-plane": "true"}, "replicas": 1, "tolerations": [{"operator": "Exists"}]}, "resources": {"limits": {"cpu": "100m", "memory": "200Mi"}, "requests": {"cpu": null, "memory": null}}, "snapshotOperator": {"controller": {"imageTag": null}, "enabled": true, "nodeSelector": {"mayadata.io/control-plane": "true"}, "provisioner": {"imageTag": null}, "replicas": 1, "tolerations": [{"operator": "Exists"}]}, "version": "1.11.0-ee"}}, "installationMode": "basic", "kind": "openebs", "links": {"cluster": "http://10.43.10.13/v3/groups/1a22/openebses/1o5/cluster", "creator": "http://10.43.10.13/v3/groups/1a22/openebses/1o5/creator", "openebsJobs": "http://10.43.10.13/v3/groups/1a22/openebses/1o5/openebsjobs", "project": "http://10.43.10.13/v3/groups/1a22/openebses/1o5/project", "remove": "http://10.43.10.13/v3/groups/1a22/openebses/1o5", "self": "http://10.43.10.13/v3/groups/1a22/openebses/1o5", "template": "http://10.43.10.13/v3/groups/1a22/openebses/1o5/template", "update": "http://10.43.10.13/v3/groups/1a22/openebses/1o5"}, "memoryResourceLimit": " ", "namespace": "openebs", "projectId": "1p3", "removed": null, "state": "creating", "templateId": "1ot2", "transitioning": "yes", "transitioningMessage": "Creating", "type": "openebs", "uuid": "6675d584-8375-462f-ad82-7a629094c396", "version": null}, "msg": "Status code was 201 and not [405]: OK (unknown bytes)", "redirected": false, "set_cookie": "PL=DirectorOnline;Path=/", "status": 201, "transfer_encoding": "chunked", "url": "http://10.43.10.13/v3/groups/1a22/clusters/1c4/openebses", "x_api_account_id": "1a22", "x_api_client_ip": "10.68.10.16", "x_api_roles": "ProjectOwner", "x_api_schemas": "http://10.43.10.13/v3/groups/1a22/schemas", "x_api_user_id": "1a21"}

amitbhatt818 commented 4 years ago

Pipeline details: https://gitlab.mayadata.io/oep/oep-e2e-konvoy/-/jobs/165862

amitbhatt818 commented 4 years ago

Test case details: https://github.com/mayadata-io/oep-e2e/blob/master/litmus/director/TCID-DIR-OP-RE-INSTALL-OPENEBS/test.yml