search

openvstorage / framework

The Framework is a set of components and tools which brings the user an interface (GUI / API) to setup, extend and manage an Open vStorage platform.
Other
27 stars 23 forks source link

Error role assignment NVME - Disk with guid '098d8066-d47a-466d-8f7c-a28ac2ef8e1a' could not be found #1046

Closed matthiasdeblock closed 7 years ago

matthiasdeblock commented 7 years ago

root@stor-01:~# apt-cache policy openvstorage openvstorage: Installed: 2.7.4-rev.4153.4979e95-1 Candidate: 2.7.4-rev.4153.4979e95-1 Version table: *\ 2.7.4-rev.4153.4979e95-1 0 500 http://apt.openvstorage.org/ unstable/main amd64 Packages 100 /var/lib/dpkg/status

Error while assigning write role to NVME. In all 4 nodes there is an NVME present. On all 4 nodes i'm getting this error:

2016-10-17 09:11:23 64900 +0200 - stor-02.cl-g8-us2 - 2398/139997937788736 - lib/disk - 68 - INFO - Disk nvme0n1 - Creating disk - {'is_ssd': True, 'name': u'nvme0n1', 
'partitions': {1048576: {'filesystem': None, 'state': 'OK', 'offset': 1048576, 'mountpoint': None, 'size': 400087318528, 'aliases': ['/dev/disk/by-partlabel/nvme0n1']}}
, 'state': 'OK', 'model': None, 'size': 400088457216, 'aliases': ['/dev/nvme0n1']}
2016-10-17 09:11:23 74500 +0200 - stor-02.cl-g8-us2 - 2398/139997937788736 - lib/scheduled tasks - 69 - INFO - Ensure single CHAINED mode - ID 1476688275_0tKGRuTyyh - T
ask ovs.disk.sync_with_reality finished successfully
2016-10-17 09:11:23 74600 +0200 - stor-02.cl-g8-us2 - 2398/139997937788736 - lib/scheduled tasks - 70 - INFO - Ensure single CHAINED mode - ID 1476688275_0tKGRuTyyh - A
mount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with_reality: 0
2016-10-17 09:11:23 75400 +0200 - stor-02.cl-g8-us2 - 2398/139997937788736 - lib/scheduled tasks - 71 - ERROR - Ensure single CHAINED mode - ID 1476688272_S9qEuRWq46 - 
Task ovs.storagerouter.configure_disk with params {'partition_guid': None, 'roles': [u'WRITE'], 'disk_guid': u'098d8066-d47a-466d-8f7c-a28ac2ef8e1a', 'storagerouter_gui
d': '50fdb03b-a75d-4c4b-b62d-11bfa726ad86', 'offset': 0, 'size': 400088457216} failed
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/storagerouter.py", line 1712, in configure_disk
    disk = Disk(disk_guid)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 233, in __init__
    self.__class__.__name__, self._guid
ObjectNotFoundException: Disk with guid '098d8066-d47a-466d-8f7c-a28ac2ef8e1a' could not be found
2016-10-17 09:11:23 75600 +0200 - stor-02.cl-g8-us2 - 2398/139997937788736 - lib/scheduled tasks - 72 - INFO - Ensure single CHAINED mode - ID 1476688272_S9qEuRWq46 - A
mount of jobs pending for key ovs_ensure_single_ovs.storagerouter.configure_disk: 0
2016-10-17 09:11:23 77400 +0200 - stor-02.cl-g8-us2 - 2093/139997937788736 - celery/celery.worker.job - 1042 - ERROR - Task ovs.storagerouter.configure_disk[e4802acc-7815-424a-9654-b0b5598ac194] raised unexpected: ObjectNotFoundException("Disk with guid '098d8066-d47a-466d-8f7c-a28ac2ef8e1a' could not be found",)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/storagerouter.py", line 1712, in configure_disk
    disk = Disk(disk_guid)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 233, in __init__
    self.__class__.__name__, self._guid
ObjectNotFoundException: Disk with guid '098d8066-d47a-466d-8f7c-a28ac2ef8e1a' could not be found

"Error" while triggering the re-scanning button in GUI:

2016-10-17 09:21:08 79800 +0200 - stor-02.cl-g8-us2 - 2398/139997937788736 - lib/disk - 113 - INFO - Disk nvme0n1 - No longer found
2016-10-17 09:21:08 81600 +0200 - stor-02.cl-g8-us2 - 2398/139997937788736 - lib/disk - 114 - INFO - Disk nvme0n1 - Deleted
kvanhijf commented 7 years ago

openvstorage-2.7.4-rev.4159.0707c77 --> https://github.com/openvstorage/framework/pull/1047

JeffreyDevloo commented 7 years ago

Steps

Output

Roles were assigned to the NVME drive:

Logs:

2016-10-20 12:02:49 61900 +0200 - ovs-esxi-host2 - 15734/140277941450560 - celery/celery.worker.strategy - 113 - INFO - Received task: ovs.storagerouter.configure_disk[1caba339-a080-44b1-b015-6f5581d71dba]
2016-10-20 12:02:49 62000 +0200 - ovs-esxi-host2 - 15734/140277941450560 - celery/celery.pool - 114 - DEBUG - TaskPool: Apply <function _fast_trace_task at 0x7f94fc4d1398> (args:('ovs.storagerouter.configure_disk', '1caba339-a080-4
4b1-b015-6f5581d71dba', ('90b8e423-e5ca-4957-b1b3-aeed0fd7ea16', u'e075b4b8-b789-4625-b642-c0f5ba1f7a4a', None, 0, 400088457216, [u'READ', u'WRITE']), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': ('9
0b8e423-e5ca-4957-b1b3-aeed0fd7ea16', u'e075b4b8-b789-4625-b642-c0f5ba1f7a4a', None, 0, 400088457216, [u'READ', u'WRITE']), 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'sr.dNhThIf6cFVTVo
Kz', u'exchange': u'generic'}, 'expires': None, u'hostname': 'celery@ovs-esxi-host2', 'task': 'ovs.storagerouter.configure_disk', 'callbacks': None, u'correlation_id': u'1caba339-a080-44b1-b015-6f5581d71dba', 'errbacks': None, 'tim
elimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'3020a37a-905b-38b6-b1fc-0870e93446a2', 'id': '1caba339-a080-44b1-b015-6f5581d71dba', u'headers': {}}) kwargs:{})
2016-10-20 12:02:49 62200 +0200 - ovs-esxi-host2 - 15734/140277941450560 - celery/celery.worker.job - 115 - DEBUG - Task accepted: ovs.storagerouter.configure_disk[1caba339-a080-44b1-b015-6f5581d71dba] pid:15759
2016-10-20 12:02:49 62400 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 62 - INFO - Ensure single CHAINED mode - ID 1476957769_mZaI5FPBTu - Amount of jobs pending for key ovs_ensure_single_ovs.storagerouter
.configure_disk: 0
2016-10-20 12:02:49 62800 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 63 - INFO - Ensure single CHAINED mode - ID 1476957769_mZaI5FPBTu - New task ovs.storagerouter.configure_disk with params {'partition_
guid': None, 'roles': [u'READ', u'WRITE'], 'disk_guid': u'e075b4b8-b789-4625-b642-c0f5ba1f7a4a', 'storagerouter_guid': '90b8e423-e5ca-4957-b1b3-aeed0fd7ea16', 'offset': 0, 'size': 400088457216} scheduled for execution
2016-10-20 12:02:49 63000 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 64 - INFO - Ensure single CHAINED mode - ID 1476957769_mZaI5FPBTu - Amount of jobs pending for key ovs_ensure_single_ovs.storagerouter
.configure_disk: 1
2016-10-20 12:02:49 63100 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 65 - INFO - Ensure single CHAINED mode - ID 1476957769_mZaI5FPBTu -   KWARGS: {'partition_guid': None, 'roles': [u'READ', u'WRITE'], '
disk_guid': u'e075b4b8-b789-4625-b642-c0f5ba1f7a4a', 'storagerouter_guid': '90b8e423-e5ca-4957-b1b3-aeed0fd7ea16', 'offset': 0, 'size': 400088457216}
2016-10-20 12:02:49 63800 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/storagerouter - 66 - DEBUG - Creating new partition - Offset: 0 bytes - Size: 400088457216 bytes - Roles: [u'READ', u'WRITE']
2016-10-20 12:02:53 02700 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 67 - INFO - Ensure single CHAINED mode - ID 1476957773_9r4yy2d150 - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_wit
h_reality: 0
2016-10-20 12:02:53 03100 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 68 - INFO - Ensure single CHAINED mode - ID 1476957773_9r4yy2d150 - New task ovs.disk.sync_with_reality with params {'storagerouter_gu
id': '90b8e423-e5ca-4957-b1b3-aeed0fd7ea16'} scheduled for execution
2016-10-20 12:02:53 03300 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 69 - INFO - Ensure single CHAINED mode - ID 1476957773_9r4yy2d150 - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_wit
h_reality: 1
2016-10-20 12:02:53 03300 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 70 - INFO - Ensure single CHAINED mode - ID 1476957773_9r4yy2d150 -   KWARGS: {'storagerouter_guid': '90b8e423-e5ca-4957-b1b3-aeed0fd7
ea16'}
2016-10-20 12:02:56 10100 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 71 - INFO - Investigating device /dev/sda
2016-10-20 12:02:56 17000 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 72 - INFO - Investigating partition /dev/sda1
2016-10-20 12:02:56 38400 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 73 - INFO - Investigating partition /dev/sda2
2016-10-20 12:02:56 52500 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 74 - INFO - Investigating partition /dev/sda3
2016-10-20 12:02:56 72100 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 75 - INFO - Investigating device /dev/sdb
2016-10-20 12:02:56 79300 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 76 - INFO - Investigating device /dev/sdc
2016-10-20 12:02:56 86500 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 77 - INFO - Investigating device /dev/sdd
2016-10-20 12:02:56 96100 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 78 - INFO - Investigating device /dev/nvme0n1
2016-10-20 12:02:57 05700 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 79 - INFO - Investigating partition /dev/nvme0n1p1
2016-10-20 12:02:57 10900 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 80 - INFO - Disk sdb - Found, updating
2016-10-20 12:02:57 12100 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 81 - INFO - Disk sda - Found, updating
2016-10-20 12:02:57 15900 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 82 - INFO - Disk sdd - Found, updating
2016-10-20 12:02:57 16700 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 83 - INFO - Disk sdc - Found, updating
2016-10-20 12:02:57 17600 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 84 - INFO - Disk nvme0n1 - Found, updating
2016-10-20 12:02:57 18300 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 85 - INFO - Disk nvme0n1 - Creating partition - {'filesystem': None, 'state': 'OK', 'offset': 1048576, 'mountpoint': None, 'size': 400087318528, 
'aliases': ['/dev/disk/by-partlabel/nvme0n1']}
2016-10-20 12:02:57 20500 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 86 - INFO - Ensure single CHAINED mode - ID 1476957773_9r4yy2d150 - Task ovs.disk.sync_with_reality finished successfully
2016-10-20 12:02:57 20800 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 87 - INFO - Ensure single CHAINED mode - ID 1476957773_9r4yy2d150 - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_wit
h_reality: 0
2016-10-20 12:02:57 21800 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/storagerouter - 88 - DEBUG - Partition created
2016-10-20 12:02:57 21900 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/storagerouter - 89 - DEBUG - Creating filesystem
2016-10-20 12:03:01 65800 +0200 - ovs-esxi-host2 - 15734/140277941450560 - celery/celery.worker.strategy - 116 - INFO - Received task: ovs.storagerouter.ping[2c24476e-a171-415f-a2a0-6fcd4c3422e9]
2016-10-20 12:03:01 66000 +0200 - ovs-esxi-host2 - 15734/140277941450560 - celery/celery.pool - 117 - DEBUG - TaskPool: Apply <function _fast_trace_task at 0x7f94fc4d1398> (args:('ovs.storagerouter.ping', '2c24476e-a171-415f-a2a0-6
fcd4c3422e9', ('90b8e423-e5ca-4957-b1b3-aeed0fd7ea16', 1476957781), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': ('90b8e423-e5ca-4957-b1b3-aeed0fd7ea16', 1476957781), 'retries': 0, u'delivery_info': 
{u'priority': 0, u'redelivered': False, u'routing_key': u'sr.dNhThIf6cFVTVoKz', u'exchange': u'generic'}, 'expires': None, u'hostname': 'celery@ovs-esxi-host2', 'task': 'ovs.storagerouter.ping', 'callbacks': None, u'correlation_id'
: u'2c24476e-a171-415f-a2a0-6fcd4c3422e9', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'6a05ca59-7fa5-3847-8e2f-42fd4952bf0d', 'id': '2c24476e-a171-415f-a2a0-6fcd4c3422e9',
 u'headers': {}}) kwargs:{})
2016-10-20 12:03:01 66200 +0200 - ovs-esxi-host2 - 15734/140277941450560 - celery/celery.worker.job - 118 - DEBUG - Task accepted: ovs.storagerouter.ping[2c24476e-a171-415f-a2a0-6fcd4c3422e9] pid:15760
2016-10-20 12:03:01 68100 +0200 - ovs-esxi-host2 - 15734/140277941450560 - celery/celery.worker.job - 119 - INFO - Task ovs.storagerouter.ping[2c24476e-a171-415f-a2a0-6fcd4c3422e9] succeeded in 0.0195507318713s: None
2016-10-20 12:03:03 57900 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 90 - INFO - Ensure single CHAINED mode - ID 1476957783_gyCQTxskON - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_wit
h_reality: 0
2016-10-20 12:03:03 58300 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 91 - INFO - Ensure single CHAINED mode - ID 1476957783_gyCQTxskON - New task ovs.disk.sync_with_reality with params {'storagerouter_gu
id': '90b8e423-e5ca-4957-b1b3-aeed0fd7ea16'} scheduled for execution
2016-10-20 12:03:03 58500 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 92 - INFO - Ensure single CHAINED mode - ID 1476957783_gyCQTxskON - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_wit
h_reality: 1
2016-10-20 12:03:03 58600 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 93 - INFO - Ensure single CHAINED mode - ID 1476957783_gyCQTxskON -   KWARGS: {'storagerouter_guid': '90b8e423-e5ca-4957-b1b3-aeed0fd7
ea16'}
2016-10-20 12:03:06 73200 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 94 - INFO - Investigating device /dev/sda
2016-10-20 12:03:06 82900 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 95 - INFO - Investigating partition /dev/sda1
2016-10-20 12:03:07 02000 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 96 - INFO - Investigating partition /dev/sda2
2016-10-20 12:03:07 13700 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 97 - INFO - Investigating partition /dev/sda3
2016-10-20 12:03:07 32500 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 98 - INFO - Investigating device /dev/sdb
2016-10-20 12:03:07 39800 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 99 - INFO - Investigating device /dev/sdc
2016-10-20 12:02:56 86500 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 77 - INFO - Investigating device /dev/sdd
2016-10-20 12:02:56 96100 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 78 - INFO - Investigating device /dev/nvme0n1
2016-10-20 12:02:57 05700 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 79 - INFO - Investigating partition /dev/nvme0n1p1
2016-10-20 12:02:57 10900 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 80 - INFO - Disk sdb - Found, updating
2016-10-20 12:02:57 12100 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 81 - INFO - Disk sda - Found, updating
2016-10-20 12:02:57 15900 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 82 - INFO - Disk sdd - Found, updating
2016-10-20 12:02:57 16700 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 83 - INFO - Disk sdc - Found, updating
2016-10-20 12:02:57 17600 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 84 - INFO - Disk nvme0n1 - Found, updating
2016-10-20 12:02:57 18300 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/disk - 85 - INFO - Disk nvme0n1 - Creating partition - {'filesystem': None, 'state': 'OK', 'offset': 1048576, 'mountpoint': None, 'size': 400087318528, 
'aliases': ['/dev/disk/by-partlabel/nvme0n1']}
2016-10-20 12:02:57 20500 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 86 - INFO - Ensure single CHAINED mode - ID 1476957773_9r4yy2d150 - Task ovs.disk.sync_with_reality finished successfully
2016-10-20 12:02:57 20800 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/scheduled tasks - 87 - INFO - Ensure single CHAINED mode - ID 1476957773_9r4yy2d150 - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_wit
h_reality: 0
2016-10-20 12:02:57 21800 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/storagerouter - 88 - DEBUG - Partition created
2016-10-20 12:02:57 21900 +0200 - ovs-esxi-host2 - 15759/140277941450560 - lib/storagerouter - 89 - DEBUG - Creating filesystem

GUI: selection_018

Test result

Roles were assigned correctly. Test passed.

Packages