Description:
When attach a cinder volume to nova host, it appeared to be successful, a device is returned. But actually the attach is not successful, 'nova show' and 'cinder show' attachment is empty.
Project:
Cinder/Nova
Project Area:
Openstack Cinder Driver/Nova
Release:
Openstack
Severity:
S1
Full Details:
create a cinder volume 'vol1'
create a nova instance 'nova1'
attach 'vol1' and 'nova1'
no error returned, but actually attach failed.
if try to detach, it will fail.
Support Material:
cmd output
stack@OpenStack-S:~/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+-------------+
| b25df735-9ed8-49d1-bd82-e7f038b6f273 | available | vol1 | 5 | lvmdriver-1 | true | False | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+-------------+
stack@OpenStack-S:~/devstack$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 6327e515-c99f-4a21-8b7b-605febe637b4 | nova1 | ACTIVE | - | Running | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+
stack@OpenStack-S:~/devstack$ nova volume-attach 6327e515-c99f-4a21-8b7b-605febe637b4 b25df735-9ed8-49d1-bd82-e7f038b6f273 auto
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb | <------------attach looks successful with device created
| id | b25df735-9ed8-49d1-bd82-e7f038b6f273 |
| serverId | 6327e515-c99f-4a21-8b7b-605febe637b4 |
| volumeId | b25df735-9ed8-49d1-bd82-e7f038b6f273 |
+----------+--------------------------------------+
stack@OpenStack-S:~/devstack$ nova volume-detach 6327e515-c99f-4a21-8b7b-605febe637b4 b25df735-9ed8-49d1-bd82-e7f038b6f273
ERROR (NotFound): volume_id not found: b25df735-9ed8-49d1-bd82-e7f038b6f273 (HTTP 404) (Request-ID: req-398d5d02-690d-424b-a9d4-3bd04f068573) <---------detach failed!
Description: When attach a cinder volume to nova host, it appeared to be successful, a device is returned. But actually the attach is not successful, 'nova show' and 'cinder show' attachment is empty.
Project: Cinder/Nova
Project Area: Openstack Cinder Driver/Nova
Release: Openstack
Severity: S1
Full Details:
Support Material:
stack@OpenStack-S:~/devstack$ nova list +--------------------------------------+-------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+------------------+ | 6327e515-c99f-4a21-8b7b-605febe637b4 | nova1 | ACTIVE | - | Running | private=10.0.0.2 | +--------------------------------------+-------+--------+------------+-------------+------------------+
stack@OpenStack-S:~/devstack$ nova volume-attach 6327e515-c99f-4a21-8b7b-605febe637b4 b25df735-9ed8-49d1-bd82-e7f038b6f273 auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | <------------attach looks successful with device created | id | b25df735-9ed8-49d1-bd82-e7f038b6f273 | | serverId | 6327e515-c99f-4a21-8b7b-605febe637b4 | | volumeId | b25df735-9ed8-49d1-bd82-e7f038b6f273 | +----------+--------------------------------------+
stack@OpenStack-S:~/devstack$ nova volume-detach 6327e515-c99f-4a21-8b7b-605febe637b4 b25df735-9ed8-49d1-bd82-e7f038b6f273 ERROR (NotFound): volume_id not found: b25df735-9ed8-49d1-bd82-e7f038b6f273 (HTTP 404) (Request-ID: req-398d5d02-690d-424b-a9d4-3bd04f068573) <---------detach failed!
2015-11-24 03:21:02.762 DEBUG nova.osapi_compute.wsgi.server req-c425bafa-b795-47bb-b854-6aa7352862b3 None None accepted ('10.108.22.98', 34012) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:826 2015-11-24 03:21:02.804 18954 DEBUG keystoneclient.auth.identity.v3.base [-] Making authentication request to http://10.108.22.98:35357/v3/auth/tokens get_auth_ref /usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py:188 2015-11-24 03:21:03.059 18954 DEBUG keystoneclient.session [-] REQ: curl -g -i --cacert "/opt/stack/data/ca-bundle.pem" -X GET http://10.108.22.98:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}1a53e19ecd13e77f5dd83360844b7896039feddb" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}0ebfbeb388c15b625e0019f1735796685c7f1f9a" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:198 2015-11-24 03:21:03.257 18954 DEBUG keystoneclient.session [-] RESP: [200] Content-Length: 4889 X-Subject-Token: {SHA1}1a53e19ecd13e77f5dd83360844b7896039feddb Vary: X-Auth-Token Keep-Alive: timeout=5, max=99 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Tue, 24 Nov 2015 08:21:03 GMT Content-Type: application/json x-openstack-request-id: req-d0c807bf-da4e-446c-a572-48fd8c669608 RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "ad8c5e36b703433b8d9f84e48a2a5bb8", "name": "anotherrole"}, {"id": "a58219baadcb4e0ea120723e7874a2ec", "name": "Member"}], "expires_at": "2015-11-24T09:21:02.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "3f45a38e06fc4c10b917162e93dbca02", "name": "demo"}, "catalog": "", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "979db8c6b97e4bfb8b07fc57cf55d805", "name": "demo"}, "audit_ids": ["CWWXUt1tRymGQkmGvbcIeQ"], "issued_at": "2015-11-24T08:21:02.735253"}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:216
2015-11-24 03:21:03.270 INFO nova.osapi_compute.wsgi.server [req-a5a31d03-f970-4595-9d90-ecca1eb9d3a9 demo demo] 10.108.22.98 "GET /v2.1/3f45a38e06fc4c10b917162e93dbca02 HTTP/1.1" status: 404 len: 264 time: 0.4905879
2015-11-24 03:21:03.302 DEBUG nova.api.openstack.wsgi [req-1cf46848-b654-41b0-a91d-6c81005a03de demo demo] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7f8ab76b0fd0>>' _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:792
2015-11-24 03:21:03.306 INFO nova.osapi_compute.wsgi.server [req-1cf46848-b654-41b0-a91d-6c81005a03de demo demo] 10.108.22.98 "GET /v2.1/ HTTP/1.1" status: 200 len: 654 time: 0.0305820
2015-11-24 03:21:24.499 DEBUG nova.osapi_compute.wsgi.server req-d7b16c60-d21d-472b-8fc8-70547ebd8c2b None None accepted ('10.108.22.98', 34024) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:826
2015-11-24 03:21:24.530 18953 DEBUG keystoneclient.session [-] REQ: curl -g -i --cacert "/opt/stack/data/ca-bundle.pem" -X GET http://10.108.22.98:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}08c40147c59eb80761cbbba0a0403239b3a6b9ce" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}39ca7b09dc8d1966cae400b7ea887b00b16a9d85" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:198
2015-11-24 03:21:24.740 18953 DEBUG keystoneclient.session [-] RESP: [200] Content-Length: 4889 X-Subject-Token: {SHA1}08c40147c59eb80761cbbba0a0403239b3a6b9ce Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Tue, 24 Nov 2015 08:21:24 GMT Content-Type: application/json x-openstack-request-id: req-8405637c-fa39-4999-9e1f-2a1740993b81
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "ad8c5e36b703433b8d9f84e48a2a5bb8", "name": "anotherrole"}, {"id": "a58219baadcb4e0ea120723e7874a2ec", "name": "Member"}], "expires_at": "2015-11-24T09:21:24.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "3f45a38e06fc4c10b917162e93dbca02", "name": "demo"}, "catalog": "", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "979db8c6b97e4bfb8b07fc57cf55d805", "name": "demo"}, "audit_ids": ["M7Nfd5noQ-mBLT4j6Lm5TA"], "issued_at": "2015-11-24T08:21:24.475525"}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:216
2015-11-24 03:21:24.754 INFO nova.osapi_compute.wsgi.server [req-4c06ced1-52a0-4cac-bbc9-3ffe0f6394f6 demo demo] 10.108.22.98 "GET /v2.1/3f45a38e06fc4c10b917162e93dbca02 HTTP/1.1" status: 404 len: 264 time: 0.2429762
2015-11-24 03:21:24.802 DEBUG nova.api.openstack.wsgi [req-1e432fa1-ea21-4ce0-b83a-04cc0162ca9e demo demo] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7f8ab76b0fd0>>' _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:792
2015-11-24 03:21:24.805 INFO nova.osapi_compute.wsgi.server [req-1e432fa1-ea21-4ce0-b83a-04cc0162ca9e demo demo] 10.108.22.98 "GET /v2.1/ HTTP/1.1" status: 200 len: 654 time: 0.0329020
2015-11-24 03:21:24.965 DEBUG nova.api.openstack.wsgi [req-7b1f2f3b-61ff-4da3-9109-d09c976f00c8 demo demo] Calling method '<bound method ServersController.show of <nova.api.openstack.compute.servers.ServersController object at 0x7f8ab7725790>>' _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:792
2015-11-24 03:21:24.969 DEBUG nova.compute.api [req-7b1f2f3b-61ff-4da3-9109-d09c976f00c8 demo demo] [instance: 6327e515-c99f-4a21-8b7b-605febe637b4] Fetching instance by UUID get /opt/stack/nova/nova/compute/api.py:2004
2015-11-24 03:21:25.166 DEBUG nova.policy [req-7b1f2f3b-61ff-4da3-9109-d09c976f00c8 demo demo] Policy check for os_compute_api:os-extended-server-attributes failed with credentials {'domain': None, 'project_name': u'demo', 'project_domain': None, 'timestamp': '2015-11-24T08:21:24.961365', 'remote_address': '10.108.22.98', 'quota_class': None, 'resource_uuid': None, 'is_admin': False, 'user': u'979db8c6b97e4bfb8b07fc57cf55d805', 'service_catalog': [{u'endpoints': [{u'adminURL': u'http://10.108.22.98:8776/v1/3f45a38e06fc4c10b917162e93dbca02', u'region': u'RegionOne', u'internalURL': u'http://10.108.22.98:8776/v1/3f45a38e06fc4c10b917162e93dbca02', u'publicURL': u'http://10.108.22.98:8776/v1/3f45a38e06fc4c10b917162e93dbca02'}], u'type': u'volume', u'name': u'cinder'}, {u'endpoints': [{u'adminURL': u'http://10.108.22.98:8776/v2/3f45a38e06fc4c10b917162e93dbca02', u'region': u'RegionOne', u'internalURL': u'http://10.108.22.98:8776/v2/3f45a38e06fc4c10b917162e93dbca02', u'publicURL': u'http://10.108.22.98:8776/v2/3f45a38e06fc4c10b917162e93dbca02'}], u'type': u'volumev2', u'name': u'cinderv2'}], 'tenant': u'3f45a38e06fc4c10b917162e93dbca02', 'read_only': False, 'project_id': u'3f45a38e06fc4c10b917162e93dbca02', 'user_id': u'979db8c6b97e4bfb8b07fc57cf55d805', 'show_deleted': False, 'roles': [u'anotherrole', u'Member'], 'user_identity': '979db8c6b97e4bfb8b07fc57cf55d805 3f45a38e06fc4c10b917162e93dbca02 - - -', 'read_deleted': 'no', 'request_id': 'req-7b1f2f3b-61ff-4da3-9109-d09c976f00c8', 'instance_lock_checked': False, 'user_domain': None, 'user_name': u'demo'} enforce /opt/stack/nova/nova/policy.py:104
2015-11-24 03:21:25.169 INFO nova.osapi_compute.wsgi.server [req-7b1f2f3b-61ff-4da3-9109-d09c976f00c8 demo demo] 10.108.22.98 "GET /v2.1/3f45a38e06fc4c10b917162e93dbca02/servers/6327e515-c99f-4a21-8b7b-605febe637b4 HTTP/1.1" status: 200 len: 1802 time: 0.2109542
2015-11-24 03:21:25.185 DEBUG nova.api.openstack.wsgi [req-398d5d02-690d-424b-a9d4-3bd04f068573 demo demo] Calling method '<bound method VolumeAttachmentController.delete of <nova.api.openstack.compute.volumes.VolumeAttachmentController object at 0x7f8ab7638190>>' _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:792
2015-11-24 03:21:25.186 DEBUG nova.compute.api [req-398d5d02-690d-424b-a9d4-3bd04f068573 demo demo] [instance: 6327e515-c99f-4a21-8b7b-605febe637b4] Fetching instance by UUID get /opt/stack/nova/nova/compute/api.py:2004
2015-11-24 03:21:25.350 DEBUG keystoneclient.session [req-398d5d02-690d-424b-a9d4-3bd04f068573 demo demo] REQ: curl -g -i -X GET http://10.108.22.98:8776/v2/3f45a38e06fc4c10b917162e93dbca02/volumes/b25df735-9ed8-49d1-bd82-e7f038b6f273 -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}08c40147c59eb80761cbbba0a0403239b3a6b9ce" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:198
2015-11-24 03:21:26.050 DEBUG keystoneclient.session [req-398d5d02-690d-424b-a9d4-3bd04f068573 demo demo] RESP: [200] Content-Length: 1341 X-Compute-Request-Id: req-9975e72e-4744-43d9-814c-327cfed3aca5 Connection: keep-alive Date: Tue, 24 Nov 2015 08:21:26 GMT Content-Type: application/json X-Openstack-Request-Id: req-9975e72e-4744-43d9-814c-327cfed3aca5
RESP BODY: {"volume": {"attachments": [], "links": [{"href": "http://10.108.22.98:8776/v2/3f45a38e06fc4c10b917162e93dbca02/volumes/b25df735-9ed8-49d1-bd82-e7f038b6f273", "rel": "self"}, {"href": "http://10.108.22.98:8776/3f45a38e06fc4c10b917162e93dbca02/volumes/b25df735-9ed8-49d1-bd82-e7f038b6f273", "rel": "bookmark"}], "availability_zone": "nova", "encrypted": false, "updated_at": "2015-11-24T08:19:27.000000", "os-volume-replication:extended_status": null, "replication_status": "disabled", "snapshot_id": null, "id": "b25df735-9ed8-49d1-bd82-e7f038b6f273", "size": 5, "user_id": "979db8c6b97e4bfb8b07fc57cf55d805", "os-vol-tenant-attr:tenant_id": "3f45a38e06fc4c10b917162e93dbca02", "metadata": {}, "status": "available", "volume_image_metadata": {"kernel_id": "27737e6b-969c-41a7-9382-ef45e76e3841", "checksum": "eb9139e4942121f22bbc2afc0400b2a4", "min_ram": "0", "ramdisk_id": "e815cdda-73bf-4cfe-944d-a4105b14c783", "disk_format": "ami", "image_name": "cirros-0.3.4-x86_64-uec", "image_id": "3d349fc1-00d2-41b6-adb5-767e37935332", "container_format": "ami", "min_disk": "0", "size": "25165824"}, "description": null, "multiattach": false, "os-volume-replication:driver_data": null, "source_volid": null, "consistencygroup_id": null, "name": "vol1", "bootable": "true", "created_at": "2015-11-24T08:14:01.000000", "volume_type": "lvmdriver-1"}}
_http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:216
2015-11-24 03:21:26.061 INFO nova.api.openstack.wsgi [req-398d5d02-690d-424b-a9d4-3bd04f068573 demo demo] HTTP exception thrown: volume_id not found: b25df735-9ed8-49d1-bd82-e7f038b6f273
2015-11-24 03:21:26.062 DEBUG nova.api.openstack.wsgi [req-398d5d02-690d-424b-a9d4-3bd04f068573 demo demo] Returning 404 to user: volume_id not found: b25df735-9ed8-49d1-bd82-e7f038b6f273 call /opt/stack/nova/nova/api/openstack/wsgi.py:1175
2015-11-24 03:21:26.064 INFO nova.osapi_compute.wsgi.server [req-398d5d02-690d-424b-a9d4-3bd04f068573 demo demo] 10.108.22.98 "DELETE /v2.1/3f45a38e06fc4c10b917162e93dbca02/servers/6327e515-c99f-4a21-8b7b-605febe637b4/os-volume_attachments/b25df735-9ed8-49d1-bd82-e7f038b6f273 HTTP/1.1" status: 404 len: 393 time: 0.8901849