Can't create partitions/set roles - permission denied

JeffreyDevloo commented 8 years ago

Problem description

When trying to configure the disks on the OVH setup, certain roles/partitions weren't set.

Choosing any different disk results in the same error. Stacktrace :(http://pastebin.com/QVSrF0sm)

2016-09-15 15:51:59 33400 +0200 - perf-stras-06 - 35005/140425817712448 - celery/celery.worker.job - 102 - DEBUG - Task accepted: ovs.storagerouter.configure_disk[141c2fcc-2181-4a90-b371-bd033ea62c71] pid:35024
2016-09-15 15:51:59 36700 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 10 - INFO - Ensure single CHAINED mode - ID 1473947519_qOng59QExV - Amount of jobs pending for key ovs_ensure_single_ovs.storagerouter.
configure_disk: 0
2016-09-15 15:51:59 39800 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 11 - INFO - Ensure single CHAINED mode - ID 1473947519_qOng59QExV - New task ovs.storagerouter.configure_disk with params {'partition_g
uid': u'708d0e6a-7cf6-4f08-9d0b-ffa0a34e3349', 'roles': [u'DB'], 'disk_guid': u'932cce70-e1d2-40ba-ae7b-bd7532f4e23f', 'storagerouter_guid': 'b86900d6-fd3a-4947-89f9-0c3856febd75', 'offset': 105394470912, 'size': 374165471232} sche
duled for execution
2016-09-15 15:51:59 42000 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 12 - INFO - Ensure single CHAINED mode - ID 1473947519_qOng59QExV - Amount of jobs pending for key ovs_ensure_single_ovs.storagerouter.
configure_disk: 1
2016-09-15 15:51:59 42000 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 13 - INFO - Ensure single CHAINED mode - ID 1473947519_qOng59QExV -   KWARGS: {'partition_guid': u'708d0e6a-7cf6-4f08-9d0b-ffa0a34e3349
', 'roles': [u'DB'], 'disk_guid': u'932cce70-e1d2-40ba-ae7b-bd7532f4e23f', 'storagerouter_guid': 'b86900d6-fd3a-4947-89f9-0c3856febd75', 'offset': 105394470912, 'size': 374165471232}
2016-09-15 15:51:59 49800 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 14 - INFO - Ensure single CHAINED mode - ID 1473947519_x2A12Jf5XY - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with
_reality: 0
2016-09-15 15:51:59 53100 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 15 - INFO - Ensure single CHAINED mode - ID 1473947519_x2A12Jf5XY - New task ovs.disk.sync_with_reality with params {'storagerouter_gui
d': 'b86900d6-fd3a-4947-89f9-0c3856febd75'} scheduled for execution
2016-09-15 15:51:59 56300 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 16 - INFO - Ensure single CHAINED mode - ID 1473947519_x2A12Jf5XY - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with
_reality: 1
2016-09-15 15:51:59 56300 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 17 - INFO - Ensure single CHAINED mode - ID 1473947519_x2A12Jf5XY -   KWARGS: {'storagerouter_guid': 'b86900d6-fd3a-4947-89f9-0c3856feb
d75'}
2016-09-15 15:52:02 82300 +0200 - perf-stras-06 - 35005/140425817712448 - celery/celery.worker.strategy - 103 - INFO - Received task: ovs.storagerouter.ping[5a0ad6a2-d622-4f35-b486-34037f08f5ce]
2016-09-15 15:52:02 82400 +0200 - perf-stras-06 - 35005/140425817712448 - celery/celery.pool - 104 - DEBUG - TaskPool: Apply <function _fast_trace_task at 0x7fb7695fc5f0> (args:('ovs.storagerouter.ping', '5a0ad6a2-d622-4f35-b486-34
037f08f5ce', ('b86900d6-fd3a-4947-89f9-0c3856febd75', 1473947522), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': ('b86900d6-fd3a-4947-89f9-0c3856febd75', 1473947522), 'retries': 0, u'delivery_info': {
u'priority': None, u'redelivered': False, u'routing_key': u'sr.kvYusXsDY8ceCIXg', u'exchange': u'generic'}, 'expires': None, u'hostname': 'celery@perf-stras-06', 'task': 'ovs.storagerouter.ping', 'callbacks': None, u'correlation_id
': u'5a0ad6a2-d622-4f35-b486-34037f08f5ce', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'e24a9fc7-d647-31f2-a3a1-a63e0e4746d1', 'id': '5a0ad6a2-d622-4f35-b486-34037f08f5ce'
, u'headers': {}}) kwargs:{})
2016-09-15 15:52:02 82600 +0200 - perf-stras-06 - 35005/140425817712448 - celery/celery.worker.job - 105 - DEBUG - Task accepted: ovs.storagerouter.ping[5a0ad6a2-d622-4f35-b486-34037f08f5ce] pid:35023
2016-09-15 15:52:03 01000 +0200 - perf-stras-06 - 35005/140425817712448 - celery/celery.worker.job - 106 - INFO - Task ovs.storagerouter.ping[5a0ad6a2-d622-4f35-b486-34037f08f5ce] succeeded in 0.184803219512s: None
2016-09-15 15:52:04 68900 +0200 - perf-stras-06 - 35024/140425817712448 - lib/disk - 18 - DEBUG - Partition /dev/sda3 has no partition metadata
2016-09-15 15:52:04 88300 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 19 - ERROR - Ensure single CHAINED mode - ID 1473947519_x2A12Jf5XY - Task ovs.disk.sync_with_reality with params {'storagerouter_guid':
 'b86900d6-fd3a-4947-89f9-0c3856febd75'} failed
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/disk.py", line 154, in sync_with_reality
    partitions_info = rem.DiskTools.get_partitions_info(match.groups()[0])
  File "/usr/lib/python2.7/dist-packages/rpyc/core/netref.py", line 196, in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
  File "/usr/lib/python2.7/dist-packages/rpyc/core/netref.py", line 71, in syncreq
    return conn.sync_request(handler, oid, *args)
  File "/usr/lib/python2.7/dist-packages/rpyc/core/protocol.py", line 441, in sync_request
    raise obj
RuntimeError: Command 'parted -m /dev/sda unit B print -s' returned non-zero exit status 1 Error: Error opening /dev/sda: Permission denied

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/tmp/tmp.Slx9HuDLTL/rpyc/core/protocol.py", line 305, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/tmp/tmp.Slx9HuDLTL/rpyc/core/protocol.py", line 535, in _handle_call
    return self._local_objects[oid](*args, **dict(kwargs))
  File "/opt/OpenvStorage/ovs/extensions/generic/disk.py", line 97, in get_partitions_info
    raise RuntimeError('{0} {1}'.format(cpe, cpe.output))
RuntimeError: Command 'parted -m /dev/sda unit B print -s' returned non-zero exit status 1 Error: Error opening /dev/sda: Permission denied
2016-09-15 15:52:04 91700 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 20 - INFO - Ensure single CHAINED mode - ID 1473947519_x2A12Jf5XY - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with_reality: 0
2016-09-15 15:52:05 04000 +0200 - perf-stras-06 - 35024/140425817712448 - lib/storagerouter - 21 - DEBUG - Using existing partition
2016-09-15 15:52:05 04100 +0200 - perf-stras-06 - 35024/140425817712448 - lib/storagerouter - 22 - DEBUG - Creating filesystem
2016-09-15 15:52:09 23900 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 23 - INFO - Ensure single CHAINED mode - ID 1473947529_AaoSg8ueVX - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with_reality: 0
2016-09-15 15:52:09 27600 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 24 - INFO - Ensure single CHAINED mode - ID 1473947529_AaoSg8ueVX - New task ovs.disk.sync_with_reality with params {'storagerouter_guid': 'b86900d6-fd3a-4947-89f9-0c3856febd75'} scheduled for execution
2016-09-15 15:52:09 30800 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 25 - INFO - Ensure single CHAINED mode - ID 1473947529_AaoSg8ueVX - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with_reality: 1
2016-09-15 15:52:09 30800 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 26 - INFO - Ensure single CHAINED mode - ID 1473947529_AaoSg8ueVX -   KWARGS: {'storagerouter_guid': 'b86900d6-fd3a-4947-89f9-0c3856febd75'}
2016-09-15 15:52:14 45800 +0200 - perf-stras-06 - 35024/140425817712448 - lib/disk - 27 - DEBUG - Partition /dev/sda3 has no partition metadata
2016-09-15 15:52:14 68300 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 28 - ERROR - Ensure single CHAINED mode - ID 1473947529_AaoSg8ueVX - Task ovs.disk.sync_with_reality with params {'storagerouter_guid': 'b86900d6-fd3a-4947-89f9-0c3856febd75'} failed
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/disk.py", line 154, in sync_with_reality
    partitions_info = rem.DiskTools.get_partitions_info(match.groups()[0])
  File "/usr/lib/python2.7/dist-packages/rpyc/core/netref.py", line 196, in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
  File "/usr/lib/python2.7/dist-packages/rpyc/core/netref.py", line 71, in syncreq
    return conn.sync_request(handler, oid, *args)
  File "/usr/lib/python2.7/dist-packages/rpyc/core/protocol.py", line 441, in sync_request
    raise obj
RuntimeError: Command 'parted -m /dev/sda unit B print -s' returned non-zero exit status 1 Error: Error opening /dev/sda: Permission denied

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/tmp/tmp.pGGoffDWqC/rpyc/core/protocol.py", line 305, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/tmp/tmp.pGGoffDWqC/rpyc/core/protocol.py", line 535, in _handle_call
    return self._local_objects[oid](*args, **dict(kwargs))
  File "/opt/OpenvStorage/ovs/extensions/generic/disk.py", line 97, in get_partitions_info
    raise RuntimeError('{0} {1}'.format(cpe, cpe.output))
RuntimeError: Command 'parted -m /dev/sda unit B print -s' returned non-zero exit status 1 Error: Error opening /dev/sda: Permission denied

2016-09-15 15:52:14 71600 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 29 - INFO - Ensure single CHAINED mode - ID 1473947529_AaoSg8ueVX - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with_reality: 0
2016-09-15 15:52:14 87200 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 30 - ERROR - Ensure single CHAINED mode - ID 1473947519_qOng59QExV - Task ovs.storagerouter.configure_disk with params {'partition_guid': u'708d0e6a-7cf6-4f08-9d0b-ffa0a34e3349', 'roles': [u'DB'], 'disk_guid': u'932cce70-e1d2-40ba-ae7b-bd7532f4e23f', 'storagerouter_guid': 'b86900d6-fd3a-4947-89f9-0c3856febd75', 'offset': 105394470912, 'size': 374165471232} failed
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/storagerouter.py", line 1844, in configure_disk
    raise RuntimeError('Unexpected filesystem')
RuntimeError: Unexpected filesystem
2016-09-15 15:52:14 89400 +0200 - perf-stras-06 - 35024/140425817712448 - lib/scheduled tasks - 31 - INFO - Ensure single CHAINED mode - ID 1473947519_qOng59QExV - Amount of jobs pending for key ovs_ensure_single_ovs.storagerouter.configure_disk: 0
2016-09-15 15:52:14 96300 +0200 - perf-stras-06 - 35005/140425817712448 - celery/celery.worker.job - 107 - INFO - Task ovs.storagerouter.configure_disk[141c2fcc-2181-4a90-b371-bd033ea62c71] succeeded in 15.628721619s: None

Additional information

Setup

Geoscale setup: ovh

Package information

openvstorage 2.7.3-rev.3958.1b99beb-1 amd64 openvStorage
openvstorage-backend 1.7.3-rev.702.e25b455-1 amd64 openvStorage Backend plugin
openvstorage-backend-core 1.7.3-rev.702.e25b455-1 amd64 openvStorage Backend plugin core
openvstorage-backend-webapps 1.7.3-rev.702.e25b455-1 amd64 openvStorage Backend plugin Web Applications
openvstorage-cinder-plugin 1.2.2-rev.38.dcc3b76-1 amd64 OpenvStorage Cinder plugin for OpenStack
openvstorage-core 2.7.3-rev.3958.1b99beb-1 amd64 openvStorage core
openvstorage-hc 1.7.3-rev.702.e25b455-1 amd64 openvStorage Backend plugin HyperConverged
openvstorage-sdm 1.6.3-rev.355.c86b45b-1 amd64 Open vStorage Backend ASD Manager
openvstorage-webapps 2.7.3-rev.3958.1b99beb-1 amd64 openvStorage Web Applications

khenderick commented 8 years ago

Most likely not fixed in previous attempt #833 for #674

khenderick commented 8 years ago

Fixed by #915, packaged in openvstorage-2.7.3-rev.3964.4c54f7e

JeffreyDevloo commented 8 years ago

The PR mentions the hostfix that was applied to resolve this issue. Verified.

openvstorage / framework