search

gluster / gcs

Check github.com/heketi, github.com/gluster/gluster-containers, or github.com/kadalu/kadalu as active alternatives
https://gluster.org
Apache License 2.0
36 stars 24 forks source link

Failed to add devices, vagrant up hung. #129

Closed kotreshhr closed 5 years ago

kotreshhr commented 5 years ago

Failed to add devices and vagrant up hung as below.

TASK [GCS | GD2 Cluster | Add devices | Add devices for kube2] *****************
Monday 04 February 2019  11:27:34 +0530 (0:00:00.097)       0:16:32.075 ******* 
FAILED - RETRYING: GCS | GD2 Cluster | Add devices | Add devices for kube2 (50 retries left).
..
...
FAILED - RETRYING: GCS | GD2 Cluster | Add devices | Add devices for kube2 (1 retries left).
failed: [kube1] (item=/dev/vdc) => {"attempts": 50, "changed": false, "connection": "close", "content": "{\"errors\":[{\"code\":1,\"message\":\"could not obtain lock: another conflicting transaction may be in progress\"}]}\n", "content_length": "110", "content_type": "application/json; charset=UTF-8", "date": "Mon, 04 Feb 2019 06:10:29 GMT", "disk": "/dev/vdc", "json": {"errors": [{"code": 1, "message": "could not obtain lock: another conflicting transaction may be in progress"}]}, "msg": "Status code was 409 and not [201]: HTTP Error 409: Conflict", "redirected": false, "status": 409, "url": "http://10.233.29.8:24007/v1/devices/062f9e10-93f5-495e-9fb4-c17e2a6fdff5", "x_gluster_cluster_id": "59aa638a-1a12-4929-854b-37097273c459", "x_gluster_peer_id": "a83e62a3-b4e8-4cbf-83a0-02b458af9fcd", "x_request_id": "fcc0bb62-97bf-4b8c-bc3f-51d7d04ddc1a"}

Following are the gd2 logs

kube1:

[vagrant@kube1 ~]$ tailf /var/log/glusterd2/glusterd2.log 
time="2019-02-04 06:15:35.756931" level=info msg="10.233.64.1 - - [04/Feb/2019:06:15:30 +0000] \"POST /v1/devices/062f9e10-93f5-495e-9fb4-c17e2a6fdff5 HTTP/1.1\" 409 110" reqid=9ceab1b3-276e-4713-be46-cc6f0a7186f7
time="2019-02-04 06:16:11.063203" level=info msg="10.233.64.1 - - [04/Feb/2019:06:16:11 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=8a93130f-6b4d-4ce2-9282-0899953cfd74
time="2019-02-04 06:16:23.131935" level=error msg="failed to obtain lock" error="could not obtain lock: another conflicting transaction may be in progress" lockID=062f9e10-93f5-495e-9fb4-c17e2a6fdff5 reqid=aa791250-8e38-4dad-9c30-43b38294f166 source="[transaction.go:75:transaction.NewTxnWithLocks]" txnid=b5ae51c1-532a-443d-8a6a-97dea59b9ddc
time="2019-02-04 06:16:23.148749" level=info msg="10.233.64.1 - - [04/Feb/2019:06:16:18 +0000] \"POST /v1/devices/062f9e10-93f5-495e-9fb4-c17e2a6fdff5 HTTP/1.1\" 409 110" reqid=aa791250-8e38-4dad-9c30-43b38294f166
time="2019-02-04 06:17:10.585921" level=error msg="failed to obtain lock" error="could not obtain lock: another conflicting transaction may be in progress" lockID=062f9e10-93f5-495e-9fb4-c17e2a6fdff5 reqid=9c3f54af-4a30-42c1-8cc2-4903b38c115c source="[transaction.go:75:transaction.NewTxnWithLocks]" txnid=9c6f2a2a-f60a-4064-917b-3a26ce07bb0b
time="2019-02-04 06:17:10.610153" level=info msg="10.233.64.1 - - [04/Feb/2019:06:17:05 +0000] \"POST /v1/devices/062f9e10-93f5-495e-9fb4-c17e2a6fdff5 HTTP/1.1\" 409 110" reqid=9c3f54af-4a30-42c1-8cc2-4903b38c115c
time="2019-02-04 06:17:11.063183" level=info msg="10.233.64.1 - - [04/Feb/2019:06:17:11 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=7798f8dc-72cf-44a2-b54a-b617105b6ba4
time="2019-02-04 06:17:57.956972" level=error msg="failed to obtain lock" error="could not obtain lock: another conflicting transaction may be in progress" lockID=062f9e10-93f5-495e-9fb4-c17e2a6fdff5 reqid=c3ffc095-5e57-4f75-914f-38659e6a0d7c source="[transaction.go:75:transaction.NewTxnWithLocks]" txnid=86bcf94d-35fc-4231-afc0-9a2c5460b9b9
time="2019-02-04 06:17:57.965078" level=info msg="10.233.64.1 - - [04/Feb/2019:06:17:52 +0000] \"POST /v1/devices/062f9e10-93f5-495e-9fb4-c17e2a6fdff5 HTTP/1.1\" 409 110" reqid=c3ffc095-5e57-4f75-914f-38659e6a0d7c

kube2:

[vagrant@kube2 ~]$ tailf /var/log/glusterd2/glusterd2.log 
time="2019-02-04 06:18:29.674176" level=error msg="failed to obtain lock" error="could not obtain lock: another conflicting transaction may be in progress" lockID=062f9e10-93f5-495e-9fb4-c17e2a6fdff5 reqid=fa763dd6-a211-424b-9f15-17c04ca187c7 source="[transaction.go:75:transaction.NewTxnWithLocks]" txnid=1ce41d5f-9bb7-4e1e-ad28-cff3b8b16af4
time="2019-02-04 06:18:29.706295" level=info msg="10.233.64.0 - - [04/Feb/2019:06:18:24 +0000] \"POST /v1/devices/062f9e10-93f5-495e-9fb4-c17e2a6fdff5 HTTP/1.1\" 409 110" reqid=fa763dd6-a211-424b-9f15-17c04ca187c7
time="2019-02-04 06:18:32.226853" level=error msg="Failed to create volume group" device=/dev/vdc error="exit status 5;   WARNING: Device /dev/vda not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/atomicos/root not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda1 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda2 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdb not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdc not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdd not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vde not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/atomicos/root not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda1 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda2 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdb not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdc not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdd not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vde not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/atomicos/root not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda1 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda2 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdb not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdc not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdd not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vde not initialized in udev database even after waiting 10000000 microseconds.\n  Physical volume '/dev/vdc' is already in volume group 'gluster-dev-vdc'\n  Unable to add physical volume '/dev/vdc' to volume group 'gluster-dev-vdc'\n  /dev/vdc: physical volume not initialized.\n" reqid=d0a875ef-2fa5-43b9-b19b-660886798dc6 source="[transaction.go:34:device.txnPrepareDevice]" txnid=c8c6ff34-2389-4798-81c6-fb627c9ffd69
time="2019-02-04 06:18:32.228935" level=error msg="Failed to create volume group" device=/dev/vdc error="exit status 5;   WARNING: Device /dev/vda not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/atomicos/root not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda1 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda2 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdb not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdc not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdd not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vde not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/atomicos/root not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda1 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda2 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdb not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdc not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdd not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vde not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/atomicos/root not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda1 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vda2 not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdb not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdc not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vdd not initialized in udev database even after waiting 10000000 microseconds.\n  WARNING: Device /dev/vde not initialized in udev database even after waiting 10000000 microseconds.\n  Physical volume '/dev/vdc' is already in volume group 'gluster-dev-vdc'\n  Unable to add physical volume '/dev/vdc' to volume group 'gluster-dev-vdc'\n  /dev/vdc: physical volume not initialized.\n  /etc/lvm/cache/.cache.tmp: rename to /etc/lvm/cache/.cache failed: No such file or directory\n" reqid=6a1bfee0-eae5-4c85-8ccb-10ace0f18a1b source="[transaction.go:34:device.txnPrepareDevice]" txnid=c5103b01-adad-48da-8078-76371ecc5341
time="2019-02-04 06:18:42.270558" level=info msg="10.233.64.0 - - [04/Feb/2019:06:02:20 +0000] \"POST /v1/devices/062f9e10-93f5-495e-9fb4-c17e2a6fdff5 HTTP/1.1\" 201 198" reqid=06a900f2-6e39-488a-b447-f85ca8828b5b
kotreshhr commented 5 years ago

@aravindavk @atinmu PTL

aravindavk commented 5 years ago

@oshankkumar ^^