mds_checkup found no preferred mds

[JeffreyDevloo]

Problem description

MDS_checkup fails due to the a mds service that got removed before another error occurred

Flow of what happenend

• Shrink vpool on .194
• Crypto error occurred from paramiko
• Vpool stuck in failure
• Set vpool to running
• Shrank vpool on the remaining nodes: .191, .192, .193
• At the last shrink/delete (.191) got the error in the GUI that certain tasks failed

Current result:

• Vpool present in the DAL.
• Vpool list in GUI not loading: the get responds with '{"error_description": "distributed_transaction_log", "error": "internal_server"}'
• Vpool config still in Arakoon

  
  In [39]: for host in Configuration.list('/ovs/vpools/3af9eba1-899c-413f-b038-e4ac78192a68/hosts/'):
  config = Configuration.get('/ovs/vpools/3af9eba1-899c-413f-b038-e4ac78192a68/hosts/{0}/config'.format(host))
  if 'distributed_transaction_log' not in config:
      print host
  ....:         

Debugging for further deletion leads to:

10.100.199.192 - ovs-workers.service: File "/opt/OpenvStorage/ovs/lib/mdsservice.py", line 793, in mds_checkup 10.100.199.192 - ovs-workers.service: mds_config_set = MDSServiceController.get_mds_storagedriver_config_set(vpool, True) 10.100.199.192 - ovs-workers.service: File "/opt/OpenvStorage/ovs/lib/mdsservice.py", line 713, in get_mds_storagedriver_config_set 10.100.199.192 - ovs-workers.service: raise RuntimeError('Could not find an MDS service')

#### Error raised due to:

mds = vp.mds_services[0] print mds.service.storagerouter.ip # 10.100.199.194 print mds.service.storagerouter_guid # 6c41a118-6f5b-4696-95b8-b2b446d874f4 for vp in VPoolList.get_vpools():
if vp.name == 'myvpool02': # vpool that got stuck break print vp.status # DELETING print vp.storagedrivers[0].storagerouter.guid # 489c9252-b50e-4828-bd57-7d6b3f40a027 print vp.storagedrivers[0].storagerouter.ip # 10.100.199.191

MDSServiceController.mds_checkup(): mds_config_set = MDSServiceController.get_mds_storagedriver_config_set(vpool, True) # <- break from here - line 809 MDSServiceController.get_mds_storagedriver_config_set(vpool, check_online=False) mds_service, load = MDSServiceController.get_preferred_mds(storagerouter, vpool) if mds_service is None: raise RuntimeError('Could not find an MDS service') # <- will break from here - line 729 MDSServiceController.get_preferred_mds(storagerouter, vpool): mds_service = (None, float('inf')) for current_mds_service in vpool.mds_services: if current_mds_service.service.storagerouter_guid == storagerouter.guid: # <- will not work in my case - line 677 load = MDSServiceController.get_mds_load(current_mds_service)[0] if mds_service is None or load < mds_service[1]: mds_service = (current_mds_service, load) return mds_service

Should we support removal of a vpool where a mds is configured on a different node than the storagedriver is?
Note that method says: preffered_mds 

#### What went wrong

StorageRouterController.remove_storagedriver: temp_client = SSHClient(sr, username='root') # Paramiko crypto errored on this point - line 922 on any storagerouter that was not .194

### Packages

ii blktap-openvstorage-utils 2.0.90-2ubuntu5 amd64 utilities to work with VHD disk images files ii libblktapctl0-openvstorage 2.0.90-2ubuntu5 amd64 Xen API blktapctl shared library (shared library) ii libvhd0-openvstorage 2.0.90-2ubuntu5 amd64 VHD file format access library ii libvhdio-2.0.90-openvstorage 2.0.90-2ubuntu5 amd64 Xen API blktap shared library (shared library) ii openvstorage 2.7.11.1-1 amd64 openvStorage ii openvstorage-backend 1.7.11.1-1 amd64 openvStorage Backend plugin ii openvstorage-backend-core 1.7.11.1-1 amd64 openvStorage Backend plugin core ii openvstorage-backend-webapps 1.7.11.1-1 amd64 openvStorage Backend plugin Web Applications ii openvstorage-core 2.7.11.1-1 amd64 openvStorage core ii openvstorage-hc 1.7.11.1-1 amd64 openvStorage Backend plugin HyperConverged ii openvstorage-health-check 3.2.0-rev.458.32b1ede-1 amd64 Open vStorage HealthCheck ii openvstorage-sdm 1.6.11.1-1 amd64 Open vStorage Backend ASD Manager ii openvstorage-webapps 2.7.11.1-1 amd64 openvStorage Web Applications

[khenderick]

1463 -> openvstorage-2.7.13-dev.1487598947.84c36f7

[khenderick]

openvstorage-2.7.13.1