search

openvstorage / framework

The Framework is a set of components and tools which brings the user an interface (GUI / API) to setup, extend and manage an Open vStorage platform.
Other
27 stars 24 forks source link

Shrink vpool fails with RuntimeError: got fault response updateClusterNodeConfigs #1007

Closed pploegaert closed 7 years ago

pploegaert commented 7 years ago 
2016-10-07 14:02:02 33200 +0200 - e191-1 - 130067/140310657390400 - celery/celery.worker.job - 427 - DEBUG - Task accepted: ovs.storagerouter.ping[42e3c069-29a3-42d0-86e9-831bf67fba78] pid:130083
2016-10-07 14:02:02 49200 +0200 - e191-1 - 130067/140310657390400 - celery/celery.worker.job - 428 - INFO - Task ovs.storagerouter.ping[42e3c069-29a3-42d0-86e9-831bf67fba78] succeeded in 0.160151725009s: None
2016-10-07 14:02:02 51400 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 218 - INFO - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Removing service volumedriver_jessica
2016-10-07 14:02:03 20000 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 219 - INFO - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Disabling service dtl_jessica
2016-10-07 14:02:03 49100 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 220 - INFO - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Stopping service dtl_jessica
2016-10-07 14:02:04 13300 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 221 - INFO - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Removing service dtl_jessica
2016-10-07 14:02:04 82600 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 222 - INFO - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Stopping service albaproxy_jessica
2016-10-07 14:02:05 04900 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 223 - INFO - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Removing service albaproxy_jessica
2016-10-07 14:02:05 43300 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 224 - INFO - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Reconfiguring volumedriver arakoon cluster
2016-10-07 14:02:05 43300 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 225 - INFO - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Node configs -
ClusterNodeConfig { vrouter_id: "jessicaalru9lFsOExLucuf", message_host: "10.100.191.31", message_port: 26200, xmlrpc_host: "10.100.191.31", xmlrpc_port: 26201, failovercache_host: "10.100.191.31", failovercache_port: 26202, network_server_uri: "tcp://127.0.0.1:26203"}
ClusterNodeConfig { vrouter_id: "jessicawoDMrvTbI9LlNUGO", message_host: "10.100.191.32", message_port: 26200, xmlrpc_host: "10.100.191.32", xmlrpc_port: 26201, failovercache_host: "10.100.191.32", failovercache_port: 26202, network_server_uri: "tcp://127.0.0.1:26203"}
2016-10-07 14:02:05 50800 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 226 - INFO - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Storage Driver 75e08e2e-f353-45cb-a6af-55b5abe52de4 jessicaalru9lFsOExLucuf - Updating cluster node configs
2016-10-07 14:02:05 64000 +0200 - e191-1 - 130084/140310657390400 - lib/storagerouter - 227 - ERROR - Remove Storage Driver - Guid 11705280-43b2-4c43-acac-e0d88e1cf82e - Reconfiguring volumedriver arakoon cluster failed
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/lib/storagerouter.py", line 1185, in remove_storagedriver
    srclient.update_cluster_node_configs(str(sd.storagedriver_id))
RuntimeError: got fault response updateClusterNodeConfigs
khenderick commented 7 years ago

Fixed by #1010, packaged in openvstorage-2.7.4-rev.4104.4a05d69

JeffreyDevloo commented 7 years ago

Information

This test is covered by our autotests.

Steps

Output

The shrinking was successful:

2016-10-18 15:45:33 98400 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.job - 317 - INFO - Task ovs.storagerouter.remove_storagedriver[0919f919-0424-4b3a-818a-160b5d0dff52] succeeded in 42.158543351s: None

Full output can be found below.

Test result

Test passed.

Full output log

016-10-18 15:44:51 82400 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.strategy - 305 - INFO - Received task: ovs.storagerouter.remove_storagedriver[0
919f919-0424-4b3a-818a-160b5d0dff52]
2016-10-18 15:44:51 82500 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.pool - 306 - DEBUG - TaskPool: Apply <function _fast_trace_task at 0x7fda95403410> (ar
gs:('ovs.storagerouter.remove_storagedriver', '0919f919-0424-4b3a-818a-160b5d0dff52', ('e89829f4-4c3c-45f0-99c4-abe81ccdcf44',), {}, {'utc': True, u'is_eager': False, 'ch
ord': None, u'group': None, 'args': ('e89829f4-4c3c-45f0-99c4-abe81ccdcf44',), 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'g
eneric.#', u'exchange': u'generic'}, 'expires': None, u'hostname': 'celery@ovs-node1', 'task': 'ovs.storagerouter.remove_storagedriver', 'callbacks': None, u'correlation_
id': u'0919f919-0424-4b3a-818a-160b5d0dff52', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'11da2d56-0e2a-3f9d-a
c4c-72f7beee27bf', 'id': '0919f919-0424-4b3a-818a-160b5d0dff52', u'headers': {}}) kwargs:{})
2016-10-18 15:44:51 82600 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 289 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Deleting Storage Driver myvpool01vVZrpKNCRNhCUt6x
2016-10-18 15:44:51 82800 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 290 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Checking availability of related Storage Routers
2016-10-18 15:44:51 82800 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.job - 307 - DEBUG - Task accepted: ovs.storagerouter.remove_storagedriver[0919f
919-0424-4b3a-818a-160b5d0dff52] pid:1965
2016-10-18 15:44:55 87000 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 291 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Available Storage Driver for migration - myvpool01xp3PGW3kCmcnn6Oe
2016-10-18 15:44:55 92500 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 292 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Storage Router ovs-node1 with IP 10.100.199.151 is online
2016-10-18 15:44:59 95600 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 293 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Available Storage Driver for migration - myvpool01vVZrpKNCRNhCUt6x
2016-10-18 15:45:00 05300 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 294 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Storage Router ovs-node2 with IP 10.100.199.152 is online
2016-10-18 15:45:00 06500 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 295 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Storage Routers on which an available Storage Driver runs: ovs-node1, ovs-node2
2016-10-18 15:45:00 06800 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 296 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Reconfiguring MDSes
2016-10-18 15:45:00 08200 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 297 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Virtual Disk e148d877-44d6-4516-ae7a-8fad14889308 myvdisk01 - Ensuring MDS safety
2016-10-18 15:45:00 08300 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 298 - DEBUG - MDS safety: vDisk e148d877-44d6-4516-ae7a-8fad14889308: Start checkup for vir
tual disk myvdisk01
2016-10-18 15:45:00 11500 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 299 - DEBUG - MDS safety: vDisk e148d877-44d6-4516-ae7a-8fad14889308: Reconfiguration required. Reasons:
2016-10-18 15:45:00 11600 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 300 - DEBUG - MDS safety: vDisk e148d877-44d6-4516-ae7a-8fad14889308:    * Slave (10.100.199.152:26300) cannot be used anymore
2016-10-18 15:45:00 15000 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 301 - DEBUG - MDS safety: vDisk e148d877-44d6-4516-ae7a-8fad14889308: Completed
2016-10-18 15:45:00 15100 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 302 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Arakoon node configs - 
ArakoonNodeConfig { node_id: "t0bcoyrSHPVSY7hF", host: "10.100.199.153", port: 26408 }
ArakoonNodeConfig { node_id: "vVZrpKNCRNhCUt6x", host: "10.100.199.152", port: 26408 }
ArakoonNodeConfig { node_id: "xp3PGW3kCmcnn6Oe", host: "10.100.199.151", port: 26408 }
2016-10-18 15:45:01 55600 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.strategy - 308 - INFO - Received task: ovs.storagerouter.ping[f79e192f-c44c-4b40-aa2a-690bc41a430e]
2016-10-18 15:45:01 55700 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.autoscale - 309 - INFO - Scaling up 1 processes.
2016-10-18 15:45:01 58800 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.pool - 310 - DEBUG - TaskPool: Apply <function _fast_trace_task at 0x7fda95403410> (args:('ovs.storagerouter.ping', 'f79e192f-c44c-4b40-aa2a-690bc41a430e', ('b3af2406-2296-498b-b403-fe89d780d561', 1476798301), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': ('b3af2406-2296-498b-b403-fe89d780d561', 1476798301), 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'sr.xp3PGW3kCmcnn6Oe', u'exchange': u'generic'}, 'expires': None, u'hostname': 'celery@ovs-node1', 'task': 'ovs.storagerouter.ping', 'callbacks': None, u'correlation_id': u'f79e192f-c44c-4b40-aa2a-690bc41a430e', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'bb98e873-6eba-3e02-8e79-9f704d3e3c20', 'id': 'f79e192f-c44c-4b40-aa2a-690bc41a430e', u'headers': {}}) kwargs:{})
2016-10-18 15:45:01 60200 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.job - 311 - DEBUG - Task accepted: ovs.storagerouter.ping[f79e192f-c44c-4b40-aa2a-690bc41a430e] pid:3801
2016-10-18 15:45:01 65000 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.job - 312 - INFO - Task ovs.storagerouter.ping[f79e192f-c44c-4b40-aa2a-690bc41a430e] succeeded in 0.0489403460015s: None
2016-10-18 15:45:02 32200 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 303 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Disabling service volumedriver_myvpool01
2016-10-18 15:45:02 59900 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 304 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Stopping service volumedriver_myvpool01
2016-10-18 15:45:04 13000 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 305 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Removing service volumedriver_myvpool01
2016-10-18 15:45:04 94200 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 306 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Disabling service dtl_myvpool01
2016-10-18 15:45:05 18700 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 307 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Stopping service dtl_myvpool01
2016-10-18 15:45:05 83700 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 308 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Removing service dtl_myvpool01
2016-10-18 15:45:06 49200 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 309 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Stopping service albaproxy_myvpool01
2016-10-18 15:45:06 81100 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 310 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Removing service albaproxy_myvpool01
2016-10-18 15:45:07 16300 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.strategy - 313 - INFO - Received task: ovs.vdisk.dtl_state_transition[793491a2-dc8d-4942-8144-cb722c04e858]
2016-10-18 15:45:07 16300 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.pool - 314 - DEBUG - TaskPool: Apply <function _fast_trace_task at 0x7fda95403410> (args:('ovs.vdisk.dtl_state_transition', '793491a2-dc8d-4942-8144-cb722c04e858', [], {'old_state': 3, 'storagedriver_id': u'myvpool01xp3PGW3kCmcnn6Oe', 'new_state': 1, 'volume_id': u'2d8c4af9-29f9-487e-8c16-8658f7dd9a0b'}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': [], 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'generic.#', u'exchange': u'generic'}, 'expires': None, u'hostname': 'celery@ovs-node1', 'task': 'ovs.vdisk.dtl_state_transition', 'callbacks': None, u'correlation_id': u'793491a2-dc8d-4942-8144-cb722c04e858', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {'old_state': 3, 'storagedriver_id': u'myvpool01xp3PGW3kCmcnn6Oe', 'new_state': 1, 'volume_id': u'2d8c4af9-29f9-487e-8c16-8658f7dd9a0b'}, 'eta': None, u'reply_to': u'8a0bdea0-fc67-3f09-bff3-491b061463c3', 'id': '793491a2-dc8d-4942-8144-cb722c04e858', u'headers': {}}) kwargs:{})
2016-10-18 15:45:07 16400 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.job - 315 - DEBUG - Task accepted: ovs.vdisk.dtl_state_transition[793491a2-dc8d-4942-8144-cb722c04e858] pid:3801
2016-10-18 15:45:07 16900 +0200 - ovs-node1 - 3801/140576860264256 - celery/celery.redirected - 311 - WARNING - 2016-10-18 15:45:07 16900 +0200 - ovs-node1 - 3801/140576860264256 - log/volumedriver_task - 310 - INFO - [ovs.lib.vdisk.dtl_state_transition] - [] - {"old_state": 3, "storagedriver_id": "myvpool01xp3PGW3kCmcnn6Oe", "new_state": 1, "volume_id": "2d8c4af9-29f9-487e-8c16-8658f7dd9a0b"} - {"storagedriver": "3e543cbd-4d8a-4789-b08e-e88a7b5c4bac"}
2016-10-18 15:45:07 17200 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.job - 316 - INFO - Task ovs.vdisk.dtl_state_transition[793491a2-dc8d-4942-8144-cb722c04e858] succeeded in 0.00813975100027s: None
2016-10-18 15:45:07 24900 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 311 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Reconfiguring volumedriver arakoon cluster
2016-10-18 15:45:07 25000 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 312 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Node configs - 
ClusterNodeConfig { vrouter_id: "myvpool01xp3PGW3kCmcnn6Oe", message_host: "10.100.199.151", message_port: 26200, xmlrpc_host: "10.100.199.151", xmlrpc_port: 26201, failovercache_host: "10.100.199.151", failovercache_port: 26202, network_server_uri: "tcp://10.100.199.151:26203"}
2016-10-18 15:45:07 25400 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 313 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Storage Driver 3e543cbd-4d8a-4789-b08e-e88a7b5c4bac myvpool01xp3PGW3kCmcnn6Oe - Updating cluster node configs
2016-10-18 15:45:07 25600 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 314 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Removing MDS services
2016-10-18 15:45:07 25600 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 315 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Remove MDS service (number 0) for Storage Router with IP 10.100.199.152
2016-10-18 15:45:11 19200 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 316 - DEBUG - Recursively removed /mnt/hdd1/myvpool01_db_mds_1
2016-10-18 15:45:11 34800 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 317 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Deleting vPool related directories and files
2016-10-18 15:45:15 25800 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 318 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Synchronizing disks with reality
2016-10-18 15:45:15 25900 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 319 - INFO - Ensure single CHAINED mode - ID 1476798315_SHAA55zgA3 - Setting initial value for key ovs_ensure_single_ovs.disk.sync_with_reality
2016-10-18 15:45:15 26100 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 320 - INFO - Ensure single CHAINED mode - ID 1476798315_SHAA55zgA3 - New task ovs.disk.sync_with_reality with params {'storagerouter_guid': '2fdf0f15-2ea4-4e04-8b2e-9ee126a02077'} scheduled for execution
2016-10-18 15:45:15 26200 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 321 - INFO - Ensure single CHAINED mode - ID 1476798315_SHAA55zgA3 - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with_reality: 1
2016-10-18 15:45:15 26200 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 322 - INFO - Ensure single CHAINED mode - ID 1476798315_SHAA55zgA3 -   KWARGS: {'storagerouter_guid': '2fdf0f15-2ea4-4e04-8b2e-9ee126a02077'}
2016-10-18 15:45:19 55200 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 323 - INFO - Investigating device /dev/sda
2016-10-18 15:45:19 63600 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 324 - INFO - Investigating device /dev/sda1
2016-10-18 15:45:19 78400 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 325 - INFO - Investigating device /dev/sdb
2016-10-18 15:45:19 84700 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 326 - INFO - Investigating device /dev/sdb1
2016-10-18 15:45:20 02600 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 327 - INFO - Investigating device /dev/sdc
2016-10-18 15:45:20 11200 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 328 - INFO - Investigating device /dev/sdc1
2016-10-18 15:45:20 35100 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 329 - INFO - Investigating device /dev/sdd
2016-10-18 15:45:20 43800 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 330 - INFO - Investigating device /dev/sdd1
2016-10-18 15:45:20 62300 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 331 - INFO - Investigating device /dev/sde
2016-10-18 15:45:20 68500 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 332 - INFO - Investigating device /dev/sde1
2016-10-18 15:45:20 87000 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 333 - INFO - Investigating device /dev/sdf
2016-10-18 15:45:20 96200 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 334 - INFO - Investigating device /dev/sdf1
2016-10-18 15:45:21 14500 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 335 - INFO - Investigating device /dev/sdg
2016-10-18 15:45:21 23200 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 336 - INFO - Investigating device /dev/sdg1
2016-10-18 15:45:21 43500 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 337 - INFO - Investigating device /dev/vda
2016-10-18 15:45:21 52000 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 338 - INFO - Investigating device /dev/vda1
2016-10-18 15:45:21 67200 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 339 - INFO - Investigating device /dev/vda2
2016-10-18 15:45:21 80900 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 340 - INFO - Investigating device /dev/vda5
2016-10-18 15:45:21 93500 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 341 - INFO - Investigating device /dev/dm-0
2016-10-18 15:45:22 07400 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 342 - INFO - Investigating device /dev/dm-1
2016-10-18 15:45:22 16200 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 343 - INFO - Disk sde - Found, updating
2016-10-18 15:45:22 17000 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 344 - INFO - Disk dm-0 - Found, updating
2016-10-18 15:45:22 17800 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 345 - INFO - Disk sdf - Found, updating
2016-10-18 15:45:22 18600 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 346 - INFO - Disk dm-1 - Found, updating
2016-10-18 15:45:22 19400 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 347 - INFO - Disk sdc - Found, updating
2016-10-18 15:45:22 31100 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 348 - INFO - Disk sdb - Found, updating
2016-10-18 15:45:22 32200 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 349 - INFO - Disk sdd - Found, updating
2016-10-18 15:45:22 33200 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 350 - INFO - Disk vda - No longer found
2016-10-18 15:45:22 34900 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 351 - INFO - Disk vda - Deleted
2016-10-18 15:45:22 34900 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 352 - INFO - Disk sdg - Found, updating
2016-10-18 15:45:22 36000 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 353 - INFO - Disk sda - Found, updating
2016-10-18 15:45:22 37000 +0200 - ovs-node1 - 1965/140576860264256 - lib/disk - 354 - INFO - Disk vda - Creating disk - {'is_ssd': False, 'name': u'vda', 'partitions': {1
048576: {'filesystem': u'ext2', 'state': 'OK', 'offset': 1048576, 'mountpoint': u'/boot', 'size': 254803968, 'aliases': ['/dev/vda1']}, 256900096: {'filesystem': None, 's
tate': 'OK', 'offset': 256900096, 'mountpoint': None, 'size': 1024, 'aliases': ['/dev/vda2']}, 256901120: {'filesystem': u'LVM2_member', 'state': 'OK', 'offset': 25690112
0, 'mountpoint': None, 'size': 25956450304, 'aliases': ['/dev/vda5']}}, 'state': 'OK', 'model': None, 'size': 26214400000, 'aliases': ['/dev/vda']}
2016-10-18 15:45:22 46100 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 355 - INFO - Ensure single CHAINED mode - ID 1476798315_SHAA55zgA3 - Task ovs.d
isk.sync_with_reality finished successfully
2016-10-18 15:45:22 46200 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 356 - INFO - Ensure single CHAINED mode - ID 1476798315_SHAA55zgA3 - Amount of 
jobs pending for key ovs_ensure_single_ovs.disk.sync_with_reality: 0
2016-10-18 15:45:22 46700 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 357 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Cleaning up model
2016-10-18 15:45:22 47200 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 358 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - 
Removing alba proxy service from model
2016-10-18 15:45:22 48400 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 359 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Checking DTL for all virtual disks in vPool myvpool01 with guid f12fde62-2c3e-43b0-9fff-c2737e5dd3d4
2016-10-18 15:45:22 48500 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 360 - INFO - Ensure single DEDUPED mode - ID 1476798322_kYBQGmYS8e - Setting key ovs_ensure_single_ovs.vdisk.dtl_checkup
2016-10-18 15:45:22 48600 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 361 - INFO - Ensure single DEDUPED mode - ID 1476798322_kYBQGmYS8e - Amount of jobs pending for key ovs_ensure_single_ovs.vdisk.dtl_checkup: 0
2016-10-18 15:45:22 48800 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 362 - INFO - Ensure single DEDUPED mode - ID 1476798322_kYBQGmYS8e - New task ovs.vdisk.dtl_checkup with params {'vpool_guid': 'f12fde62-2c3e-43b0-9fff-c2737e5dd3d4'} scheduled for execution
2016-10-18 15:45:22 48900 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 363 - INFO - Ensure single DEDUPED mode - ID 1476798322_kYBQGmYS8e - Amount of jobs pending for key ovs_ensure_single_ovs.vdisk.dtl_checkup: 1
2016-10-18 15:45:22 48900 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 364 - INFO - Ensure single DEDUPED mode - ID 1476798322_kYBQGmYS8e -   KWARGS: {'vpool_guid': 'f12fde62-2c3e-43b0-9fff-c2737e5dd3d4'}
2016-10-18 15:45:22 49300 +0200 - ovs-node1 - 1965/140576860264256 - lib/vdisk - 365 - INFO - DTL checkup started
2016-10-18 15:45:22 49600 +0200 - ovs-node1 - 1965/140576860264256 - lib/vdisk - 366 - INFO -     Verifying vDisk myvdisk01 with guid e148d877-44d6-4516-ae7a-8fad14889308
2016-10-18 15:45:24 55600 +0200 - ovs-node1 - 1965/140576860264256 - extensions/servicemanager - 367 - INFO - Service ovs-dtl_myvpool01 could not be found.
2016-10-18 15:45:24 55800 +0200 - ovs-node1 - 1965/140576860264256 - lib/vdisk - 368 - WARNING -     DTL service on Storage Router with IP 10.100.199.152 is not reachable
2016-10-18 15:45:24 56100 +0200 - ovs-node1 - 1965/140576860264256 - lib/vdisk - 369 - INFO -     No Storage Routers could be found as valid DTL target, setting DTL for vDisk to STANDALONE
2016-10-18 15:45:24 56600 +0200 - ovs-node1 - 1965/140576860264256 - lib/vdisk - 370 - INFO - DTL checkup ended
2016-10-18 15:45:24 66200 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 371 - INFO - Ensure single DEDUPED mode - ID 1476798322_kYBQGmYS8e - Task ovs.vdisk.dtl_checkup finished successfully
disk.dtl_checkup finished successfully
2016-10-18 15:45:24 66400 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 372 - INFO - Ensure single DEDUPED mode - ID 1476798322_kYBQGmYS8e - Amount of jobs pending for key ovs_ensure_single_ovs.vdisk.dtl_checkup: 0
2016-10-18 15:45:24 68100 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagerouter - 373 - INFO - Remove Storage Driver - Guid e89829f4-4c3c-45f0-99c4-abe81ccdcf44 - Running MDS checkup
2016-10-18 15:45:24 68200 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 374 - INFO - Ensure single CHAINED mode - ID 1476798324_xy5ibn5Xtn - Setting initial value for key ovs_ensure_single_ovs.mds.mds_checkup
2016-10-18 15:45:24 68400 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 375 - INFO - Ensure single CHAINED mode - ID 1476798324_xy5ibn5Xtn - New task ovs.mds.mds_checkup with default params scheduled for execution
2016-10-18 15:45:24 68500 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 376 - INFO - Ensure single CHAINED mode - ID 1476798324_xy5ibn5Xtn - Amount of jobs pending for key ovs_ensure_single_ovs.mds.mds_checkup: 1
2016-10-18 15:45:24 68500 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 377 - INFO - Ensure single CHAINED mode - ID 1476798324_xy5ibn5Xtn -   KWARGS: {}
2016-10-18 15:45:24 68800 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 378 - INFO - MDS checkup - Started
2016-10-18 15:45:24 69100 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 379 - INFO - MDS checkup - vPool myvpool02
2016-10-18 15:45:26 32700 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 380 - INFO - MDS checkup - vPool myvpool02 - Storage Router ovs-node1 - ONLINE
2016-10-18 15:45:27 83100 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 381 - INFO - MDS checkup - vPool myvpool02 - Storage Router ovs-node2 - ONLINE
2016-10-18 15:45:27 83200 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 382 - INFO - MDS checkup - vPool myvpool01
2016-10-18 15:45:29 48800 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 383 - INFO - MDS checkup - vPool myvpool01 - Storage Router ovs-node1 - ONLINE
2016-10-18 15:45:29 49000 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 384 - INFO - MDS checkup - vPool myvpool02 - Storage Router ovs-node1 - Capacity available: True
2016-10-18 15:45:29 49100 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 385 - INFO - MDS checkup - vPool myvpool02 - Storage Router ovs-node2 - Capacity available: True
2016-10-18 15:45:31 19800 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 386 - INFO - MDS checkup - vPool myvpool02 - Storage Router ovs-node1 - Storing default MDS configuration: [{'host': u'10.100.199.151', 'port': 26301}]
2016-10-18 15:45:31 20100 +0200 - ovs-node1 - 1965/140576860264256 - extensions/storagedriver - 387 - DEBUG - No need to apply changes, nothing changed
2016-10-18 15:45:31 20100 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagedriver - 387 - DEBUG - No need to apply changes, nothing changed
2016-10-18 15:45:31 20600 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 388 - INFO - MDS checkup - vPool myvpool02 - Storage Router ovs-node2 - Storing default MDS configuration: [{'host': u'10.100.199.152', 'port': 26301}]
2016-10-18 15:45:31 21100 +0200 - ovs-node1 - 1965/140576860264256 - extensions/storagedriver - 389 - INFO - Applying storagedriver configuration changes on 10.100.199.152
2016-10-18 15:45:31 21100 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagedriver - 389 - INFO - Applying storagedriver configuration changes on 10.100.199.152
2016-10-18 15:45:33 75100 +0200 - ovs-node1 - 1965/140576860264256 - extensions/storagedriver - 390 - INFO - Changed fs_metadata_backend_mds_nodes from "[mds://10.100.199.151:26301]" to "[mds://10.100.199.152:26301]"
2016-10-18 15:45:33 75100 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagedriver - 390 - INFO - Changed fs_metadata_backend_mds_nodes from "[mds://10.100.199.151:26301]" to "[mds://10.100.199.152:26301]"
2016-10-18 15:45:33 75200 +0200 - ovs-node1 - 1965/140576860264256 - extensions/storagedriver - 391 - INFO - Changes applied
2016-10-18 15:45:33 75200 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagedriver - 391 - INFO - Changes applied
2016-10-18 15:45:33 75200 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 392 - INFO - MDS checkup - vPool myvpool02 - Ensuring safety for all virtual disks
2016-10-18 15:45:33 75400 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 393 - INFO - MDS checkup - vPool myvpool01 - Storage Router ovs-node1 - Capacity available: True
2016-10-18 15:45:33 77900 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 394 - INFO - MDS checkup - vPool myvpool01 - Storage Router ovs-node1 - Storing default MDS configuration: [{'host': u'10.100.199.151', 'port': 26300}]
2016-10-18 15:45:33 78200 +0200 - ovs-node1 - 1965/140576860264256 - extensions/storagedriver - 395 - DEBUG - No need to apply changes, nothing changed
2016-10-18 15:45:33 78200 +0200 - ovs-node1 - 1965/140576860264256 - lib/storagedriver - 395 - DEBUG - No need to apply changes, nothing changed
2016-10-18 15:45:33 78200 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 396 - INFO - MDS checkup - vPool myvpool01 - Ensuring safety for all virtual disks
2016-10-18 15:45:33 78400 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 397 - DEBUG - MDS safety: vDisk e148d877-44d6-4516-ae7a-8fad14889308: Start checkup for virtual disk myvdisk01
2016-10-18 15:45:33 81500 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 398 - DEBUG - MDS safety: vDisk e148d877-44d6-4516-ae7a-8fad14889308: No reconfiguration required
2016-10-18 15:45:33 82400 +0200 - ovs-node1 - 1965/140576860264256 - lib/mds - 399 - INFO - MDS checkup - Finished
2016-10-18 15:45:33 97000 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 400 - INFO - Ensure single CHAINED mode - ID 1476798324_xy5ibn5Xtn - Task ovs.mds.mds_checkup finished successfully
2016-10-18 15:45:33 97200 +0200 - ovs-node1 - 1965/140576860264256 - lib/scheduled tasks - 401 - INFO - Ensure single CHAINED mode - ID 1476798324_xy5ibn5Xtn - Amount of jobs pending for key ovs_ensure_single_ovs.mds.mds_checkup: 0
2016-10-18 15:45:33 98400 +0200 - ovs-node1 - 26383/140576860264256 - celery/celery.worker.job - 317 - INFO - Task ovs.storagerouter.remove_storagedriver[0919f919-0424-4b3a-818a-160b5d0dff52] succeeded in 42.158543351s: None

Packages