search

mayadata-io / oep-e2e

Plan tests and run them for OpenEBS Enterprise Platform
Apache License 2.0
6 stars 10 forks source link

Cstor pool recommendation jobs are failing #605

Closed amitbhatt818 closed 3 years ago

amitbhatt818 commented 4 years ago
shovanmaity commented 4 years ago

More detailed log required to debug this problem. Required info of all the BD present in the system(size, state, claim state, hostname, file system ... ) before creating a cStor pool.

shovanmaity commented 4 years ago

Note: The requirements are -

shovanmaity commented 4 years ago

Please provide these details if we have or try to add them so that I can debug it.

amitbhatt818 commented 4 years ago

Openebs version= openebs.io/version=1.10.0 Total no of blockdevice = 9 Size of all Bds is same All the BDs are in unclaimed and Active state File systems are not present in BDs.

NAME                                           NODENAME                                   SIZE          CLAIMSTATE   STATUS   AGE
blockdevice-1447fcbe83f614345678bfee657aa718   ip-10-0-1-226.us-west-2.compute.internal   21474836480   Claimed      Active   15m
blockdevice-346a39ef7bd37708e2fd6fa73d350959   ip-10-0-1-226.us-west-2.compute.internal   21474836480   Claimed      Active   15m
blockdevice-4a6b9ad532d7198158443d510fa01350   ip-10-0-1-226.us-west-2.compute.internal   21474836480   Unclaimed    Active   15m
blockdevice-552de9f520c63ae6c60a12c3cdb6c0bd   ip-10-0-1-226.us-west-2.compute.internal   21474836480   Unclaimed    Active   15m
blockdevice-5e75dae9139d9f52aad4f9b9d8c2f16a   ip-10-0-1-226.us-west-2.compute.internal   21474836480   Unclaimed    Active   15m
blockdevice-60b27441f77bdb68e3a6cb6dd9335101   ip-10-0-1-226.us-west-2.compute.internal   21474836480   Unclaimed    Active   15m
blockdevice-6963dcba924386d69d71d4ae0846cdd9   ip-10-0-1-226.us-west-2.compute.internal   21474836480   Unclaimed    Active   15m
blockdevice-8c6c691c843326dde46c0679974a84b4   ip-10-0-1-226.us-west-2.compute.internal   21474836480   Unclaimed    Active   15m
blockdevice-c6cf4fd5416289d434212e808de9636d   ip-10-0-1-226.us-west-2.compute.internal   21474836480   Unclaimed    Active   15m
shovanmaity commented 4 years ago

@amitbhatt818 I meant I need it in the job log while it was failing at that time.

amitbhatt818 commented 4 years ago

I guess configuration will be same for all the pipelines. But then also I will wait for the pipeline to get fail and then I will paste the logs here. @shovanmaity

amitbhatt818 commented 4 years ago

Raised PR to get details of BD before pool creation. I need to discover the issue again in pipeline. I will paste the log of job when it will fail again. https://github.com/mayadata-io/oep-e2e/pull/611/ cc: @shovanmaity @AmitKumarDas

amitbhatt818 commented 4 years ago

Logs of failed job with BD details: @shovanmaity 

TASK [Get username] ************************************************************
changed: [127.0.0.1] => {"changed": true, "cmd": "cat /etc/secret-volume/username", "delta": "0:00:00.001910", "end": "2020-05-27 07:13:59.643121", "rc": 0, "start": "2020-05-27 07:13:59.641211", "stderr": "", "stderr_lines": [], "stdout": "6162AE606B863A0B6A01", "stdout_lines": ["6162AE606B863A0B6A01"]}

TASK [Get password] ************************************************************
changed: [127.0.0.1] => {"changed": true, "cmd": "cat /etc/secret-volume/password", "delta": "0:00:00.005923", "end": "2020-05-27 07:13:59.859882", "rc": 0, "start": "2020-05-27 07:13:59.853959", "stderr": "", "stderr_lines": [], "stdout": "J4bB29ctDgVNBTYovf1rc7XWCb6UxmHsk1yRWxvt", "stdout_lines": ["J4bB29ctDgVNBTYovf1rc7XWCb6UxmHsk1yRWxvt"]}

TASK [Check whether openebs components are in Running state or not] ************
changed: [127.0.0.1] => (item=openebs-provisioner) => {"ansible_loop_var": "item", "attempts": 1, "changed": true, "cmd": "kubectl get pods -n openebs  | grep openebs-provisioner | awk '{print $3}' | awk -F':' '{print $1}' | tail -n 1", "delta": "0:00:00.069846", "end": "2020-05-27 07:14:00.148403", "item": "openebs-provisioner", "rc": 0, "start": "2020-05-27 07:14:00.078557", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}
changed: [127.0.0.1] => (item=openebs-ndm-operator) => {"ansible_loop_var": "item", "attempts": 1, "changed": true, "cmd": "kubectl get pods -n openebs  | grep openebs-ndm-operator | awk '{print $3}' | awk -F':' '{print $1}' | tail -n 1", "delta": "0:00:00.074597", "end": "2020-05-27 07:14:00.368329", "item": "openebs-ndm-operator", "rc": 0, "start": "2020-05-27 07:14:00.293732", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}
changed: [127.0.0.1] => (item=openebs-ndm) => {"ansible_loop_var": "item", "attempts": 1, "changed": true, "cmd": "kubectl get pods -n openebs  | grep openebs-ndm | awk '{print $3}' | awk -F':' '{print $1}' | tail -n 1", "delta": "0:00:00.060294", "end": "2020-05-27 07:14:00.574893", "item": "openebs-ndm", "rc": 0, "start": "2020-05-27 07:14:00.514599", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}
changed: [127.0.0.1] => (item=openebs-snapshot-operator) => {"ansible_loop_var": "item", "attempts": 1, "changed": true, "cmd": "kubectl get pods -n openebs  | grep openebs-snapshot-operator | awk '{print $3}' | awk -F':' '{print $1}' | tail -n 1", "delta": "0:00:00.058239", "end": "2020-05-27 07:14:00.776683", "item": "openebs-snapshot-operator", "rc": 0, "start": "2020-05-27 07:14:00.718444", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}
changed: [127.0.0.1] => (item=openebs-admission-server) => {"ansible_loop_var": "item", "attempts": 1, "changed": true, "cmd": "kubectl get pods -n openebs  | grep openebs-admission-server | awk '{print $3}' | awk -F':' '{print $1}' | tail -n 1", "delta": "0:00:00.060098", "end": "2020-05-27 07:14:00.982549", "item": "openebs-admission-server", "rc": 0, "start": "2020-05-27 07:14:00.922451", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}
changed: [127.0.0.1] => (item=openebs-localpv-provisioner) => {"ansible_loop_var": "item", "attempts": 1, "changed": true, "cmd": "kubectl get pods -n openebs  | grep openebs-localpv-provisioner | awk '{print $3}' | awk -F':' '{print $1}' | tail -n 1", "delta": "0:00:00.061121", "end": "2020-05-27 07:14:01.188945", "item": "openebs-localpv-provisioner", "rc": 0, "start": "2020-05-27 07:14:01.127824", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}
changed: [127.0.0.1] => (item=maya-apiserver) => {"ansible_loop_var": "item", "attempts": 1, "changed": true, "cmd": "kubectl get pods -n openebs  | grep maya-apiserver | awk '{print $3}' | awk -F':' '{print $1}' | tail -n 1", "delta": "0:00:00.071520", "end": "2020-05-27 07:14:01.405752", "item": "maya-apiserver", "rc": 0, "start": "2020-05-27 07:14:01.334232", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}

TASK [Get BD] ******************************************************************
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get bd -n openebs\n", "delta": "0:00:00.059139", "end": "2020-05-27 07:14:01.681406", "rc": 0, "start": "2020-05-27 07:14:01.622267", "stderr": "", "stderr_lines": [], "stdout": "NAME                                           NODENAME                                  SIZE          CLAIMSTATE   STATUS   AGE\nblockdevice-220a3a84614ecbf7c9ffd5cc0fa21d51   ip-10-0-1-52.us-west-2.compute.internal   21474836480   Unclaimed    Active   2m16s\nblockdevice-6f10fae0757daf0d783ca29704235ba3   ip-10-0-1-52.us-west-2.compute.internal   21474836480   Unclaimed    Active   2m16s", "stdout_lines": ["NAME                                           NODENAME                                  SIZE          CLAIMSTATE   STATUS   AGE", "blockdevice-220a3a84614ecbf7c9ffd5cc0fa21d51   ip-10-0-1-52.us-west-2.compute.internal   21474836480   Unclaimed    Active   2m16s", "blockdevice-6f10fae0757daf0d783ca29704235ba3   ip-10-0-1-52.us-west-2.compute.internal   21474836480   Unclaimed    Active   2m16s"]}

TASK [Fetch recommendations details] *******************************************
ok: [127.0.0.1] => {"changed": false, "connection": "close", "content": "{\"type\":\"collection\",\"resourceType\":\"recommendation\",\"links\":{\"self\":\"http:\\/\\/54.202.216.241\\/v3\\/groups\\/1a20\\/recommendations\"},\"createTypes\":{},\"actions\":{},\"data\":[{\"id\":\"1r!cStorPool\",\"type\":\"recommendation\",\"links\":{\"self\":\"http:\\/\\/54.202.216.241\\/v3\\/groups\\/1a20\\/recommendations\\/1r!cstorpool\"},\"actions\":{\"getcapacityrecommendation\":\"http:\\/\\/54.202.216.241\\/v3\\/groups\\/1a20\\/recommendations\\/1r!cstorpool\\/?action=getcapacityrecommendation\",\"getdevicerecommendation\":\"http:\\/\\/54.202.216.241\\/v3\\/groups\\/1a20\\/recommendations\\/1r!cstorpool\\/?action=getdevicerecommendation\"},\"baseType\":\"recommendation\",\"name\":\"cStorPool\",\"description\":\"Get recommendation for cStor pool\"}],\"sortLinks\":{},\"pagination\":null,\"sort\":null,\"filters\":{},\"createDefaults\":{}}", "content_length": "767", "content_type": "application/json; charset=utf-8", "cookies": {"PL": "DirectorOnline"}, "cookies_string": "PL=DirectorOnline", "date": "Wed, 27 May 2020 07:14:02 GMT", "elapsed": 0, "expires": "Thu, 01 Jan 1970 00:00:00 GMT", "json": {"actions": {}, "createDefaults": {}, "createTypes": {}, "data": [{"actions": {"getcapacityrecommendation": "http://54.202.216.241/v3/groups/1a20/recommendations/1r!cstorpool/?action=getcapacityrecommendation", "getdevicerecommendation": "http://54.202.216.241/v3/groups/1a20/recommendations/1r!cstorpool/?action=getdevicerecommendation"}, "baseType": "recommendation", "description": "Get recommendation for cStor pool", "id": "1r!cStorPool", "links": {"self": "http://54.202.216.241/v3/groups/1a20/recommendations/1r!cstorpool"}, "name": "cStorPool", "type": "recommendation"}], "filters": {}, "links": {"self": "http://54.202.216.241/v3/groups/1a20/recommendations"}, "pagination": null, "resourceType": "recommendation", "sort": null, "sortLinks": {}, "type": "collection"}, "msg": "OK (767 bytes)", "redirected": false, "set_cookie": "PL=DirectorOnline;Path=/", "status": 200, "url": "http://54.202.216.241/v3/groups/1a20/recommendations", "vary": "Accept-Encoding, Accept-Encoding, User-Agent", "x_api_account_id": "1a20", "x_api_client_ip": "54.188.125.153", "x_api_roles": "ProjectOwner", "x_api_schemas": "http://54.202.216.241/v3/groups/1a20/schemas", "x_api_user_id": "1a19"}

TASK [Fetch the recommendation id] *********************************************
ok: [127.0.0.1] => {"ansible_facts": {"recommendation_id": "1r!cStorPool"}, "changed": false}

TASK [List Recommendations] ****************************************************
ok: [127.0.0.1] => {"changed": false, "connection": "close", "content": "{\"type\":\"collection\",\"resourceType\":\"recommendation\",\"links\":{\"self\":\"http:\\/\\/54.202.216.241\\/v3\\/groups\\/1a20\\/recommendations\\/1r!cStorPool\\/\"},\"createTypes\":{},\"actions\":{},\"data\":[],\"sortLinks\":{},\"pagination\":null,\"sort\":null,\"filters\":{},\"createDefaults\":{}}", "content_length": "265", "content_type": "application/json; charset=utf-8", "cookies": {"PL": "DirectorOnline"}, "cookies_string": "PL=DirectorOnline", "date": "Wed, 27 May 2020 07:14:02 GMT", "elapsed": 0, "expires": "Thu, 01 Jan 1970 00:00:00 GMT", "json": {"actions": {}, "createDefaults": {}, "createTypes": {}, "data": [], "filters": {}, "links": {"self": "http://54.202.216.241/v3/groups/1a20/recommendations/1r!cStorPool/"}, "pagination": null, "resourceType": "recommendation", "sort": null, "sortLinks": {}, "type": "collection"}, "msg": "OK (265 bytes)", "redirected": false, "set_cookie": "PL=DirectorOnline;Path=/", "status": 200, "url": "http://54.202.216.241/v3/groups/1a20/recommendations/1r!cStorPool/?action=getcapacityrecommendation", "vary": "Accept-Encoding", "x_api_account_id": "1a20", "x_api_client_ip": "54.188.125.153", "x_api_roles": "ProjectOwner", "x_api_schemas": "http://54.202.216.241/v3/groups/1a20/schemas", "x_api_user_id": "1a19"}

TASK [Fetch the deviceGroupName] ***********************************************
fatal: [127.0.0.1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: list object has no element 0\n\nThe error appears to be in '/litmus/director/TCID-DIR-OP-CSTOR-POOL-RECOMMEND-CREATE-MIRROR/test.yml': line 77, column 11, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n        ## Fetch the deviceGroupName\n        - name: Fetch the deviceGroupName\n          ^ here\n"}

TASK [Setting fail flag] *******************************************************
ok: [127.0.0.1] => {"ansible_facts": {"flag": "Fail"}, "changed": false}

TASK [include_tasks] ***********************************************************
included: /ansible-utils/update_litmus_result_resource.yml for 127.0.0.1

TASK [Generate the litmus result CR to reflect SOT (Start of Test)] ************
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [Analyze the cr yaml] *****************************************************
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [Apply the litmus result CR] **********************************************
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [Generate the litmus result CR to reflect EOT (End of Test)] **************
changed: [127.0.0.1] => {"changed": true, "checksum": "c7e5d5da8d4852e73f719e222d859ec34e2f6ca4", "dest": "./litmus-result.yaml", "gid": 0, "group": "root", "md5sum": "306376736c3c4705e3e6874a1a58e431", "mode": "0644", "owner": "root", "size": 421, "src": "/root/.ansible/tmp/ansible-tmp-1590563642.9924002-421-21958237562809/source", "state": "file", "uid": 0}

TASK [Analyze the cr yaml] *****************************************************
changed: [127.0.0.1] => {"changed": true, "cmd": "cat litmus-result.yaml", "delta": "0:00:00.002159", "end": "2020-05-27 07:14:03.494558", "rc": 0, "start": "2020-05-27 07:14:03.492399", "stderr": "", "stderr_lines": [], "stdout": "---\napiVersion: litmus.io/v1alpha1\nkind: LitmusResult\nmetadata:\n\n  # name of the litmus testcase\n  name: create-cstor-pool-mirror \nspec:\n\n  # holds information on the testcase\n  testMetadata:\n    app:  \n    chaostype:  \n\n  # holds the state of testcase,  manually updated by json merge patch\n  # result is the useful value today, but anticipate phase use in future \n  testStatus: \n    phase: completed  \n    result: Fail ", "stdout_lines": ["---", "apiVersion: litmus.io/v1alpha1", "kind: LitmusResult", "metadata:", "", "  # name of the litmus testcase", "  name: create-cstor-pool-mirror ", "spec:", "", "  # holds information on the testcase", "  testMetadata:", "    app:  ", "    chaostype:  ", "", "  # holds the state of testcase,  manually updated by json merge patch", "  # result is the useful value today, but anticipate phase use in future ", "  testStatus: ", "    phase: completed  ", "    result: Fail "]}

TASK [Apply the litmus result CR] **********************************************
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl apply -f litmus-result.yaml", "delta": "0:00:00.147416", "end": "2020-05-27 07:14:03.849647", "failed_when_result": false, "rc": 0, "start": "2020-05-27 07:14:03.702231", "stderr": "", "stderr_lines": [], "stdout": "litmusresult.litmus.io/create-cstor-pool-mirror configured", "stdout_lines": ["litmusresult.litmus.io/create-cstor-pool-mirror configured"]}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=18   changed=10   unreachable=0    failed=0    skipped=8    rescued=1    ignored=0   

++ kubectl get litmusresult create-cstor-pool-mirror --no-headers -o custom-columns=:spec.testStatus.result
+ testResult=Fail
+ echo Fail
Fail
+ '[' Fail '!=' Pass ']'
+ bash utils/e2e-cr jobname:tcid-dir-op-cstor-pool-recommend-create-mirror jobphase:Completed
++ cut -d : -f 2
++ echo jobname:tcid-dir-op-cstor-pool-recommend-create-mirror
+ job_name=tcid-dir-op-cstor-pool-recommend-create-mirror
++ cut -d : -f 2
++ echo jobphase:Completed
+ job_phase=Completed
++ cut -d : -f2-
++ echo
+ test_result=
+ echo '******Creating e2e-result-custom-resource********'
******Creating e2e-result-custom-resource********
+ [[ Completed == \W\a\i\t\i\n\g ]]
+ [[ Completed == \R\u\n\n\i\n\g ]]
+ echo hi2
hi2
+ cat utils/e2e-result.yml
+ sed -i -e 's/<test-phase>/Completed/g' result-tcid-dir-op-cstor-pool-recommend-create-mirror.yml
+ sed -i -e 's/<test-name>/tcid-dir-op-cstor-pool-recommend-create-mirror/g' result-tcid-dir-op-cstor-pool-recommend-create-mirror.yml
+ cat result-tcid-dir-op-cstor-pool-recommend-create-mirror.yml
---
apiVersion: e2e.io/v1alpha1
kind: e2eResult
metadata:

  # name of the e2e testcase
  name: tcid-dir-op-cstor-pool-recommend-create-mirror
spec:
  testResult: <test-result>
  testStatus:
    phase: Completed
    build_id: <openebs-id>   
+ '[' '!' -z '' ']'
+ sed -i 's/<openebs-id>/0.8.1/g' result-tcid-dir-op-cstor-pool-recommend-create-mirror.yml
+ cat result-tcid-dir-op-cstor-pool-recommend-create-mirror.yml
---
apiVersion: e2e.io/v1alpha1
kind: e2eResult
metadata:

  # name of the e2e testcase
  name: tcid-dir-op-cstor-pool-recommend-create-mirror
spec:
  testResult: <test-result>
  testStatus:
    phase: Completed
    build_id: 0.8.1   
+ kubectl apply -f result-tcid-dir-op-cstor-pool-recommend-create-mirror.yml
e2eresult.e2e.io/tcid-dir-op-cstor-pool-recommend-create-mirror configured
+ exit 1
Running after_script
Uploading artifacts for failed job
ERROR: Job failed: exit code 1
amitbhatt818 commented 4 years ago

Pipeline detail: https://gitlab.mayadata.io/oep/oep-e2e-aws/-/jobs/130161

shovanmaity commented 4 years ago

@amitbhatt818 can you point me to the test yaml?

amitbhatt818 commented 4 years ago

https://github.com/mayadata-io/oep-e2e/blob/master/litmus/director/TCID-DIR-OP-CSTOR-POOL-RECOMMEND-CREATE-MIRROR/test.yml

shovanmaity commented 4 years ago

It looks like a setup issue after having a debug session with @amitbhatt818. cc @AmitKumarDas @ajeshbaby @avishnu is it good to include system checks before running test case?

Problem - File system was present in the disks.

AmitKumarDas commented 4 years ago

@sonasingh46 @shovanmaity is CSPC not able to detect the same?

shovanmaity commented 4 years ago

@AmitKumarDas as the disks has file system it was not giving the recommendation. which is correct only for that it was not able to create a pool and the job failed. So for system configuration the pool creation failed.

shovanmaity commented 4 years ago

I think we should have a requirement check before running a test case. ie - If I want to create a stripe pool then I should have one formated disk attached(not mounted) to my node.

AmitKumarDas commented 4 years ago

@shovanmaity @amitbhatt818 Let me put my question in this way. Will a user be able to understand what has gone wrong? IMO this is much more than adding checks before running tests.

sonasingh46 commented 4 years ago

is CSPC not able to detect the same?

@AmitKumarDas -- CSPC does not detect anything. It creates the cStor CRs and deployments(pool-manager) Now, if a block device has file-system present on it -- pool creation will fail. And this fails is intentional as we do not want to override a filesystem as it could have some potential data. We do not wipe the data rather expect users to give a fresh block device. Just a precautionary measure to avoid data loss.

shovanmaity commented 4 years ago

Will a user be able to understand what has gone wrong? IMO this is much more than adding checks before running tests.

Yes. If user uses cli (kubectl apply) then the admission webhook will give an error. If user uses director then he/she will not get any recommendation.

And in both cases there will be an error message.

cc @AmitKumarDas

AmitKumarDas commented 4 years ago

@shovanmaity @amitbhatt818 what are the exact error messages.

shovanmaity commented 4 years ago

In webhook user will get file system present in the disk. In director user will get not have enough eligible disk to create pool.

shovanmaity commented 4 years ago

System validation is much required here As in the statement, it was saying the system has 9 disks but actually there was 2 disks. We are trying to create both stripe and mirror pool so at least 3 eligible disks required.

amitbhatt818 commented 4 years ago

Yes I am working on it.Thanks, @shovanmaity @AmitKumarDas.

amitbhatt818 commented 4 years ago

In AWS sometime file system is getting created and sometime it is not getting created in the disk. When file system is not created in the disk at that time jobs are getting passed. https://gitlab.mayadata.io/oep/oep-e2e-aws/pipelines/10897

amit-bhatt@Amit-Bhatt:~$ kubectl get bd -n openebs -ojsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.kubernetes\.io/hostname}{"\t"}{.status}{"\t"}{.spec.filesystem}{"\t"}{"\n"}{end}'
blockdevice-1039f06e5e11481a3ae310a86d3b75da    ip-10-0-1-186.us-west-2.compute.internal    map[claimState:Claimed state:Active]    map[]   
blockdevice-57431d9034d652f84b2de3f3228aded8    ip-10-0-1-186.us-west-2.compute.internal    map[claimState:Claimed state:Active]    map[]   
blockdevice-60addaace2e3788b2e06ab4865e7f67c    ip-10-0-1-186.us-west-2.compute.internal    map[claimState:Unclaimed state:Active]  map[]   
blockdevice-66f0a9b300f9eef10fd9f9cbf6b76127    ip-10-0-1-186.us-west-2.compute.internal    map[claimState:Unclaimed state:Active]  map[]   
blockdevice-7719f1993c8a758f8d7550c9762a9127    ip-10-0-1-186.us-west-2.compute.internal    map[claimState:Unclaimed state:Active]  map[]   
blockdevice-7e145a494e927ccc44ffd8b96b5cdd47    ip-10-0-1-186.us-west-2.compute.internal    map[claimState:Unclaimed state:Active]  map[]   
blockdevice-804d61a1e7944a4af0822f75d5ceca14    ip-10-0-1-186.us-west-2.compute.internal    map[claimState:Unclaimed state:Active]  map[]   
blockdevice-e3c6d0ff58fab49a77780e5548c5da0f    ip-10-0-1-186.us-west-2.compute.internal    map[claimState:Unclaimed state:Active]  map[]   
blockdevice-ff8b2ca504c20f366764b00aa4297f63    ip-10-0-1-186.us-west-2.compute.internal    map[claimState:Unclaimed state:Active]  map[]

And when file system is being created in the disk pipeline is failing. https://gitlab.mayadata.io/oep/oep-e2e-aws/pipelines/10892

amit-bhatt@Amit-Bhatt:~$ kubectl get bd -n openebs -ojsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.kubernetes\.io/hostname}{"\t"}{.status}{"\t"}{.spec.filesystem}{"\t"}{"\n"}{end}'
blockdevice-220a3a84614ecbf7c9ffd5cc0fa21d51    ip-10-0-1-52.us-west-2.compute.internal map[claimState:Unclaimed state:Active]  map[fsType:ext4 mountPoint:/mnt/master-vol-0be0d4bb31d6fc711]   
blockdevice-6f10fae0757daf0d783ca29704235ba3    ip-10-0-1-52.us-west-2.compute.internal map[claimState:Unclaimed state:Active]  map[fsType:ext4 mountPoint:/mnt/master-vol-085a9fe2bf7035b86]

@nrusinko working on it to make it consistent in AWS. He need to make few changes in his script so that we can avoid file system creation in the disk. cc: @ajeshbaby @shovanmaity @AmitKumarDas