S3: on minio vm robot error, we need to kick start healing

[muhamadazmy]

• It happens that due to a vdisk failure, the robot crashes before minio actually detects the error on the disk. In that case we need to kick start the self healing (based on minio role either active or passive)

• We need to make sure to handle both internal server errors, or down time (robot not reachable)

[Pishoy]

verified and self healing is trigger successfully

• vm data before disk failure

  Out[10]: 
  {'data': {'configs': [],
  'cpu': 2,
  'disks': [{'diskType': 'ssd', 'label': 's3vm', 'size': 10}],
  'image': 'zero-os',
  'kernelArgs': [{'key': 'development', 'name': 'development'},
  {'key': 'zerotier', 'name': 'zerotier', 'value': '159924d63085650f'},
  {'key': 'ztid',
  'name': 'ztid',
  'value': '17dbaa0f78:0:e5e4eaf40763b09848c28b23b71d83a1ade0c25461c42ecdfd96a04f5708d554ec5359e0f0fc81d27b1ae3a5ef4ca0f53eed3a30c340d95bf0b5a82803ce8845:840089f1d2f8e882eb9ad98a1251fcdae5deb0bcc7605898f0c940718df641bfa3bef18811a3baa64a5deca0c77a2bdede70df1c8c479979e126d9c8bfbc7d11'}],
  'memory': 4000,
  'mgmtNic': {'id': '159924d63085650f',
  'type': 'zerotier',
  'ztClient': 'client'},
  'nodeId': 'ac1f6b2723e0',
  'ports': [{'name': 'minio_25b907ed-927c-461c-b89d-f35a9d44a08b',
  'source': 1043,
  'target': 1024}],
  'ztIdentity': '17dbaa0f78:0:e5e4eaf40763b09848c28b23b71d83a1ade0c25461c42ecdfd96a04f5708d554ec5359e0f0fc81d27b1ae3a5ef4ca0f53eed3a30c340d95bf0b5a82803ce8845:840089f1d2f8e882eb9ad98a1251fcdae5deb0bcc7605898f0c940718df641bfa3bef18811a3baa64a5deca0c77a2bdede70df1c8c479979e126d9c8bfbc7d11'},
  'guid': '7764cb09-ca13-407f-a596-e842d9a1229f',
  'name': '25b907ed-927c-461c-b89d-f35a9d44a08b',
  'public': False,
  'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'},
  {'category': 'status', 'state': 'ok', 'tag': 'running'}],
  'template': 'github.com/threefoldtech/0-templates/dm_vm/0.0.1',
  'version': '0.0.1'}

• vm data after self healing

In [122]: vmpassive.data Out[122]: {'data': {'configs': [], 'cpu': 2, 'disks': [{'diskType': 'ssd', 'label': 's3vm', 'size': 10}], 'image': 'zero-os', 'kernelArgs': [{'key': 'development', 'name': 'development'}, {'key': 'zerotier', 'name': 'zerotier', 'value': '159924d63085650f'}, {'key': 'ztid', 'name': 'ztid', 'value': '69397d739a:0:784930ca6b23915b3d249f7fc6b5690da01ff702205abe944f4fc1b0c15cf578d6de4a74c42bd75a0d90c8e3dae5154f0c184560a3c358b77c9cd005484f67ba:991b8fe144de9924c6ef52f097db0b45476fd014be286c249405a848c7cf41eb25eaa1c485631245e59b2136bab84a336e9ca182e602d21e98203101ea3d2cee'}], 'memory': 4000, 'mgmtNic': {'id': '159924d63085650f', 'type': 'zerotier', 'ztClient': 'client'}, 'nodeId': 'ac1f6b272370', 'ports': [{'name': 'minio_25b907ed-927c-461c-b89d-f35a9d44a08b', 'source': 1041, 'target': 1024}], 'ztIdentity': '69397d739a:0:784930ca6b23915b3d249f7fc6b5690da01ff702205abe944f4fc1b0c15cf578d6de4a74c42bd75a0d90c8e3dae5154f0c184560a3c358b77c9cd005484f67ba:991b8fe144de9924c6ef52f097db0b45476fd014be286c249405a848c7cf41eb25eaa1c485631245e59b2136bab84a336e9ca182e602d21e98203101ea3d2cee'}, 'guid': '2b289357-827a-48c9-b699-ab38c093c52c', 'name': '25b907ed-927c-461c-b89d-f35a9d44a08b', 'public': False, 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'}, {'category': 'status', 'state': 'ok', 'tag': 'running'}], 'template': 'github.com/threefoldtech/0-templates/dm_vm/0.0.1', 'version': '0.0.1'}