Open kotreshhr opened 5 years ago
Glusterd2 logs from the node.
time="2019-02-12 08:20:49.381749" level=info msg="10.233.65.7 - - [12/Feb/2019:08:20:49 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=1b2aad2e-8832-402d-ba38-65e103e4368c
time="2019-02-12 08:20:50.794884" level=info msg="client connected" address="10.233.66.13:49151" server=sunrpc source="[server.go:148:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2019-02-12 08:20:50.833154" level=info msg="client connected" address="10.233.66.13:1023" server=sunrpc source="[server.go:148:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2019-02-12 08:20:50.834550" level=info msg="client disconnected" address="10.233.66.13:1023" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2019-02-12 08:21:19.686324" level=info msg="10.233.66.1 - - [12/Feb/2019:08:21:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=c8b6d0e3-bc00-4505-a238-cb287842707b
time="2019-02-12 08:22:19.685364" level=info msg="10.233.66.1 - - [12/Feb/2019:08:22:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=0e4c2478-f1bd-4606-bd5c-aae38bd6bbff
time="2019-02-12 08:23:19.685525" level=info msg="10.233.66.1 - - [12/Feb/2019:08:23:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=775dbb45-df76-495f-bdc7-c0c688d3a080
time="2019-02-12 08:23:53.008514" level=info msg="10.233.65.7 - - [12/Feb/2019:08:23:52 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-1125e02f-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=f9f3c2cd-9b93-4308-a502-e58a84eba7f7
time="2019-02-12 08:23:53.176259" level=info msg="10.233.65.7 - - [12/Feb/2019:08:23:52 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-112b6aad-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=fb185961-72b3-4b4c-8828-11526cde54ab
time="2019-02-12 08:23:54.100340" level=info msg="10.233.65.7 - - [12/Feb/2019:08:23:54 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11583441-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=16324faf-5d22-43c2-b779-7488c26a6ec2
time="2019-02-12 08:23:54.140917" level=info msg="10.233.65.7 - - [12/Feb/2019:08:23:53 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-1141c3d8-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=cc2c830c-6eac-4ab9-b19b-0323296b1a57
time="2019-02-12 08:24:03.124718" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:02 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-1181dd86-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=95d28272-8e98-4d44-9526-822088d11f4d
time="2019-02-12 08:24:03.406890" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:02 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11b0f5da-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=df1d6326-ef20-44bc-9bda-ed3e35f610d2
time="2019-02-12 08:24:03.535776" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:03 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11ebc52e-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=3fcd115b-d806-447a-b4f6-c1b300458fb0
time="2019-02-12 08:24:04.647908" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:03 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11c11bf7-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=6d009f9d-848d-4222-8769-5a789223f428
time="2019-02-12 08:24:08.641742" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:08 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-114d88b3-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=ad62b11a-830a-41d9-9bcc-84d81de6f740
time="2019-02-12 08:24:18.640362" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:18 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11ebc52e-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=275e0c15-bc41-4e40-882a-e2188b78be4a
time="2019-02-12 08:24:19.686095" level=info msg="10.233.66.1 - - [12/Feb/2019:08:24:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=a0765567-7877-4368-959e-128908542306
time="2019-02-12 08:24:20.213604" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:16 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=dc41a749-db21-4db6-a6f8-9154952bcc45
time="2019-02-12 08:24:21.300029" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:19 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=35200ef0-d8bb-45ed-b0f4-1d67998e9b86
time="2019-02-12 08:24:22.313401" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:20 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=ed40dd32-a2e1-4ba6-861e-4b3281704d6a
time="2019-02-12 08:24:23.479884" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:21 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=1898b399-d284-4d4e-a433-90df2f899d40
time="2019-02-12 08:24:24.005789" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:22 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=aec69878-a68f-479d-83a2-1519afaf901a
time="2019-02-12 08:24:25.362623" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:25 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=e3c6ef2b-6750-4b5b-985a-0165f0fa2713
time="2019-02-12 08:24:26.047034" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:23 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=27891918-b0b9-4835-ba54-3b13d294e84f
time="2019-02-12 08:24:26.236353" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:24 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=cca5d5d7-56c1-4258-abed-94025ea2f89b
time="2019-02-12 08:24:26.369764" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:26 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=a76de0f8-ad18-4ec5-910b-817d1e983122
time="2019-02-12 08:24:28.743582" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:27 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=6c734a35-aebf-42c1-8c29-256ea9ba09ae
time="2019-02-12 08:24:28.748878" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:28 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=cf65a502-f05f-475c-832d-de47937574d4
time="2019-02-12 08:24:29.150398" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:29 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=14c16d6a-0a3c-4a68-8231-297106ea76af
time="2019-02-12 08:24:35.280816" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:30 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=e85ddda3-bd43-4789-8739-eceb17b9b4e2
time="2019-02-12 08:24:38.258086" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:35 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=a3c57e7b-5a9f-44a0-8bb7-3d9407e98826
time="2019-02-12 08:24:38.283329" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:38 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=a8f6fe97-ecfa-4bc4-9f34-9bc87042813d
time="2019-02-12 08:24:48.739426" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:48 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11ebc52e-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=5c4baa30-6efb-4563-94ff-8e4c58d5e652
time="2019-02-12 08:25:16.821361" level=error msg="error in acquiring cluster lock" error="could not obtain lock: another conflicting transaction may be in progress" source="[volume.go:94:utils.ResizeBlockHostingVolume]"
time="2019-02-12 08:25:16.825462" level=error msg="failed in updating hostvolume _block-hosting-available-size metadata" block_name=pvc-96704fe9-2e9f-11e9-ab2f-525400995719 error="could not obtain lock: another conflicting transaction may be in progress" hostvol=block_hosting_volume_c0e4d154-0f93-41d7-8408-32e1e5ff61ba requested_block_size=104857600 source="[gluster_virtblock.go:110:gluster-virtblock.(*GlusterVirtBlk).CreateBlockVolume]"
time="2019-02-12 08:25:16.825636" level=info msg="10.233.65.7 - - [12/Feb/2019:08:25:11 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 500 110" reqid=efc63e35-21c9-49a7-a1ff-47a8493558ec
time="2019-02-12 08:25:19.687135" level=info msg="10.233.66.1 - - [12/Feb/2019:08:25:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=dde1646a-aa47-4aed-8e35-e754c92d351f
time="2019-02-12 08:25:44.537604" level=info msg="10.233.65.7 - - [12/Feb/2019:08:25:44 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-114d88b3-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=700c518c-afa1-4ff3-9599-81f26e738a84
time="2019-02-12 08:26:19.687209" level=info msg="10.233.66.1 - - [12/Feb/2019:08:26:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=c840a6c6-6997-4526-85f7-8e454609b837
time="2019-02-12 08:26:51.733639" level=error msg="error in acquiring cluster lock" error="could not obtain lock: another conflicting transaction may be in progress" source="[volume.go:94:utils.ResizeBlockHostingVolume]"
time="2019-02-12 08:26:51.733703" level=error msg="failed in updating hostvolume _block-hosting-available-size metadata" block_name=pvc-96a4dd44-2e9f-11e9-ab2f-525400995719 error="could not obtain lock: another conflicting transaction may be in progress" hostvol=block_hosting_volume_c0e4d154-0f93-41d7-8408-32e1e5ff61ba requested_block_size=104857600 source="[gluster_virtblock.go:110:gluster-virtblock.(*GlusterVirtBlk).CreateBlockVolume]"
time="2019-02-12 08:26:51.733758" level=info msg="10.233.65.7 - - [12/Feb/2019:08:26:46 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 500 110" reqid=4554eac7-7aab-46dd-9d47-d1e3739dd5f9
time="2019-02-12 08:27:19.687883" level=info msg="10.233.66.1 - - [12/Feb/2019:08:27:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=ea07874f-63d5-469d-9cd5-fd6c067a5e85
time="2019-02-12 08:27:44.577533" level=info msg="10.233.65.7 - - [12/Feb/2019:08:27:44 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-114d88b3-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=881eedda-f39b-49ac-97f0-7454baa39dd4
time="2019-02-12 08:28:19.685340" level=info msg="10.233.66.1 - - [12/Feb/2019:08:28:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=7e3dc719-53f5-4070-913a-3dd36211b835
time="2019-02-12 08:29:03.178720" level=error msg="error in acquiring cluster lock" error="could not obtain lock: another conflicting transaction may be in progress" source="[volume.go:94:utils.ResizeBlockHostingVolume]"
time="2019-02-12 08:29:03.178919" level=error msg="failed in updating hostvolume _block-hosting-available-size metadata" block_name=pvc-96704fe9-2e9f-11e9-ab2f-525400995719 error="could not obtain lock: another conflicting transaction may be in progress" hostvol=block_hosting_volume_c0e4d154-0f93-41d7-8408-32e1e5ff61ba requested_block_size=104857600 source="[gluster_virtblock.go:110:gluster-virtblock.(*GlusterVirtBlk).CreateBlockVolume]"
time="2019-02-12 08:29:03.179350" level=info msg="10.233.65.7 - - [12/Feb/2019:08:28:57 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 500 110" reqid=a3dbf8ca-29c4-42c6-b7cc-30b6b2d72001
time="2019-02-12 08:29:19.687312" level=info msg="10.233.66.1 - - [12/Feb/2019:08:29:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=52f2e59c-731a-47f1-ae53-d30cb498d8fe
Is it the same on latest container? Did we get any further ?