search

openvstorage / framework-alba-plugin

The Framework ALBA plugin extends the OpenvStorage GUI with functionality to manage ASDs (Alternate Storage Daemon) and Seagate Kinetic drives.
Other
2 stars 3 forks source link

Too many open files #599

Open openvstorage-ci opened 6 years ago

openvstorage-ci commented 6 years ago

From @jeroenmaelbrancke on February 13, 2018 8:35

Can you add a proper error state for the 'too many open files' error and let us know which node suffers from this?

root@SV4SRV0021:~# ovs healthcheck alba backend-test
[INFO] Storagerouter Id: KjVk832AslAaVwZ4
[INFO] Environment Os: Ubuntu 16.04 xenial
[INFO] Hostname: SV4SRV0021
[INFO] Cluster Id: JoQXf3AYDGVMBZtb
[INFO] Storagerouter Type: MASTER
[INFO] Starting OpenvStorage Healthcheck version 3.4.5-1
[INFO] ======================
[INFO] Checking available ALBA backends.
Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/OpenvStorage/ovs/dal/hybrids/albabackend.py", line 101, in _load_live_info
    _data = _node.storage_stack
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 348, in <lambda>
    fget = lambda s: s._get_dynamic_property(dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 417, in _get_dynamic_property
    return self._backend_property(data_loader, dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 1030, in _backend_property
    dynamic_data = fct()
  File "/opt/OpenvStorage/ovs/dal/hybrids/albanode.py", line 63, in _storage_stack
    disk_data = self.client.get_disks()
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 137, in get_disks
    return self._call(requests.get, 'disks', clean=True)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 107, in _call
    raise RuntimeError(error_message)
RuntimeError: [Errno 24] Too many open files: '/dev/disk'

Exception in thread Thread-38:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/OpenvStorage/ovs/dal/hybrids/albabackend.py", line 101, in _load_live_info
    _data = _node.storage_stack
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 348, in <lambda>
    fget = lambda s: s._get_dynamic_property(dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 417, in _get_dynamic_property
    return self._backend_property(data_loader, dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 1030, in _backend_property
    dynamic_data = fct()
  File "/opt/OpenvStorage/ovs/dal/hybrids/albanode.py", line 63, in _storage_stack
    disk_data = self.client.get_disks()
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 137, in get_disks
    return self._call(requests.get, 'disks', clean=True)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 107, in _call
    raise RuntimeError(error_message)
RuntimeError: [Errno 24] Too many open files: '/dev/disk'

No handlers could be found for logger "ovs_extensions.generic.volatilemutex"
Exception in thread Thread-57:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/OpenvStorage/ovs/dal/hybrids/albabackend.py", line 101, in _load_live_info
    _data = _node.storage_stack
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 348, in <lambda>
    fget = lambda s: s._get_dynamic_property(dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 417, in _get_dynamic_property
    return self._backend_property(data_loader, dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 1030, in _backend_property
    dynamic_data = fct()
  File "/opt/OpenvStorage/ovs/dal/hybrids/albanode.py", line 63, in _storage_stack
    disk_data = self.client.get_disks()
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 137, in get_disks
    return self._call(requests.get, 'disks', clean=True)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 107, in _call
    raise RuntimeError(error_message)
RuntimeError: [Errno 24] Too many open files: '/dev/disk'

Exception in thread Thread-83:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/OpenvStorage/ovs/dal/hybrids/albabackend.py", line 101, in _load_live_info
    _data = _node.storage_stack
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 348, in <lambda>
    fget = lambda s: s._get_dynamic_property(dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 417, in _get_dynamic_property
    return self._backend_property(data_loader, dynamic)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 1030, in _backend_property
    dynamic_data = fct()
  File "/opt/OpenvStorage/ovs/dal/hybrids/albanode.py", line 63, in _storage_stack
    disk_data = self.client.get_disks()
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 137, in get_disks
    return self._call(requests.get, 'disks', clean=True)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 107, in _call
    raise RuntimeError(error_message)
RuntimeError: [Errno 24] Too many open files: '/dev/disk'

[SUCCESS] We found 5 backend(s)!
[INFO] Checking the ALBA ASDs.
[INFO] Checking separate ASDs for backend sv4-hddbackend03:
[SUCCESS] ASD test with DISK_ID L7jO0m3ROyabs9OMyaWvJkJV0hPjF8XV succeeded!
[SUCCESS] ASD test with DISK_ID zJ0TTqWbMWs1xV2hF3IdG1m570JXP3kQ succeeded!
[SUCCESS] ASD test with DISK_ID 15XUfbDfB9Q4meTJWEi87QotX7KRnyJy succeeded!
[SUCCESS] ASD test with DISK_ID keV4fmSxSRjzr8ODfRjezyfOyqDH48qO succeeded!
[SUCCESS] ASD test with DISK_ID sGugAUYqDUPo0c0kZbJszxsgWCFapKvn succeeded!
[SUCCESS] ASD test with DISK_ID YC904ZjtBpAsVk1TFVeRQID5r33WbbgG succeeded!
[WARNING] ASD test with DISK_ID PWaZTXaCoRXIPjaC4j1D5OzxoK4i2dvR failed because: nodedown
[WARNING] ASD test with DISK_ID ePqC4Srjy27angOiNmGn3jDuKEe7fpuP failed because: nodedown
[WARNING] ASD test with DISK_ID r8KuO7zE6jhglz04Q9PYzewt6ILIi0kp failed because: nodedown
[SUCCESS] ASD test with DISK_ID GEXsOxgzBoNPZZ0wqqVNRSEntnO4HeT3 succeeded!
[SUCCESS] ASD test with DISK_ID 3r5TcVBXN0DNjupiD4MVL8XZofvHdF5v succeeded!
[SUCCESS] ASD test with DISK_ID t2GMTsZAhXFtFkKydXCQ9iq9dV7gpfhQ succeeded!
[SUCCESS] ASD test with DISK_ID p4JO1OfBjfQ08ViGqhLBW1GNILln92AW succeeded!
[SUCCESS] ASD test with DISK_ID bSioAHQ5BreyKml1oE7Y0S3LmD14CRIE succeeded!
[SUCCESS] ASD test with DISK_ID j4ivKa7oFJXHLZif7BJeK5bZeFQ3FiJe succeeded!
[SUCCESS] ASD test with DISK_ID 6RXyP1kI9rydffOFMzCMIfz7XgbqoZw2 succeeded!
[SUCCESS] ASD test with DISK_ID pOAv0WFxleQHWiISTD14YZdkiZFqbuIf succeeded!
[SUCCESS] ASD test with DISK_ID gl2dVxGWJv7uSp5fQouYSFTjDb347OTC succeeded!
[SUCCESS] ASD test with DISK_ID o4KJ3iuH3RRLUP1FXBLhbskGr5N7rRf8 succeeded!
[SUCCESS] ASD test with DISK_ID EWLF7kdcdW6r4MnLHCNhbZUYSkYhNWFr succeeded!
[SUCCESS] ASD test with DISK_ID 9jBcEGVBRMoIuV6SkhS2nCOCfjOp2tMY succeeded!
[SUCCESS] ASD test with DISK_ID cWQc77Atk65plNxFfKGFD76FRUmvI4gM succeeded!
[SUCCESS] ASD test with DISK_ID uX6FwieXQU1CKQtSQYvIbBlccuftaewt succeeded!
[SUCCESS] ASD test with DISK_ID RMGp2z74qkt4QzzBqK9bZuM2lwYgckfi succeeded!
[SUCCESS] ASD test with DISK_ID cTIwN0SViK2epSHiI5JqbeDQVSY298nM succeeded!
[SUCCESS] ASD test with DISK_ID 6HBGz0RmO1N9ZRueLlnXJPvTV8WR5oEu succeeded!
[SUCCESS] ASD test with DISK_ID 95dAjfjUy99Nq4CTLuLYZyOjSgNoHWNQ succeeded!
[SUCCESS] ASD test with DISK_ID WcHHnOI1rw6DfajjGZZaMBPJiriojtWk succeeded!
[WARNING] ASD test with DISK_ID Xpo5IzC1q0xmHWml3OyQvGml6iXOaxJF failed because: unknown
[WARNING] ASD test with DISK_ID i1nE6N1j5qPBkfMMROqGemFd9F5pNveU failed because: unknown
[WARNING] ASD test with DISK_ID FeniKSyezD0pqgSnHKPtoQyGxVIL9KrP failed because: unknown
[WARNING] ASD test with DISK_ID kN510d0JXbyL9uXJLtpQ403ScigJ8L7D failed because: unknown
[WARNING] ASD test with DISK_ID UH5dEOW5DwgPKDn4Mm785J58JKuypeek failed because: unknown
[WARNING] ASD test with DISK_ID XWxgvUk9vaZ1gDiQzBzV45FG0Gyldlta failed because: unknown
[WARNING] Alba backend sv4-hddbackend03 should be available for VPool use with 25 asds, but there are 9 defective asds: PWaZTXaCoRXIPjaC4j1D5OzxoK4i2dvR, ePqC4Srjy27angOiNmGn3jDuKEe7fpuP, r8KuO7zE6jhglz04Q9PYzewt6ILIi0kp, Xpo5IzC1q0xmHWml3OyQvGml6iXOaxJF, i1nE6N1j5qPBkfMMROqGemFd9F5pNveU, FeniKSyezD0pqgSnHKPtoQyGxVIL9KrP, kN510d0JXbyL9uXJLtpQ403ScigJ8L7D, UH5dEOW5DwgPKDn4Mm785J58JKuypeek, XWxgvUk9vaZ1gDiQzBzV45FG0Gyldlta
[INFO] Checking separate ASDs for backend sv4-hddbackend02:
[SUCCESS] ASD test with DISK_ID Jobp5M99EvZ02gYIRGJ0nh6DPsQyFujA succeeded!
[SUCCESS] ASD test with DISK_ID tL0IJ5QOMTINnaVvfAJsGuP8tTtMmdjM succeeded!
[SUCCESS] ASD test with DISK_ID wCJ2EmJiRM9cIaAiD0PWu7ReABYiCnW2 succeeded!
[SUCCESS] ASD test with DISK_ID ugCH4EB4aGML5uViVrFOG7rtrSsLHBlf succeeded!
[SUCCESS] ASD test with DISK_ID Fdh8R8GtpQNlqV3Wq0CdWl7oNyNM3Wjt succeeded!
[SUCCESS] ASD test with DISK_ID VcrrKUhaLOOGI7bTbovVnzd1jccqM7Vr succeeded!
[WARNING] ASD test with DISK_ID JSlAW8XvG2GDzGGZNgXMkO8cdYooKEZd failed because: nodedown
[WARNING] ASD test with DISK_ID rvl7pYCsJxi0kTCgVRQKsx9zZ1w0P540 failed because: nodedown
[WARNING] ASD test with DISK_ID S3KkojHnGMq8mp6Rz8LtFY9mi6jGqFFg failed because: nodedown
[SUCCESS] ASD test with DISK_ID sr4JcRvu9mVY6qWVrXE4wnDXiupdxYMt succeeded!
[SUCCESS] ASD test with DISK_ID k66UmwP9DpFnTM3GKvgbO85UsnnzU72c succeeded!
[SUCCESS] ASD test with DISK_ID P8kh8WhTgVydWZOq97nLxJfMeA8j06WJ succeeded!
[SUCCESS] ASD test with DISK_ID F4Pdx4nQOBqM67DeBJh9NzUmBmdeqgxt succeeded!
[SUCCESS] ASD test with DISK_ID oMa65aKlADJw8rCsq5y8MXWVGpIn24mB succeeded!
[SUCCESS] ASD test with DISK_ID DWf27znUjm0hJRfPQeP2lP2m9dIjKgi1 succeeded!
[SUCCESS] ASD test with DISK_ID UngscLez1PW2jqonmgTprtdbN6mwv00N succeeded!
[SUCCESS] ASD test with DISK_ID UAKvS4u2F9yrADfZW1PtxHL9w2DireWf succeeded!
[SUCCESS] ASD test with DISK_ID fRmvSancb471LzIqkkfc9wYv8eQgq5zl succeeded!
[SUCCESS] ASD test with DISK_ID kmc0eRkyANrSkBcsldNS0N4Q7oDIuaXt succeeded!
[SUCCESS] ASD test with DISK_ID tbhhmO7Tj2PKkRIM4Hg2HnbRRkdRrfJU succeeded!
[SUCCESS] ASD test with DISK_ID AbKKEFAHYTnyvnmQvl1dJRlUsUv6I1JB succeeded!
[SUCCESS] ASD test with DISK_ID Dvj3STv8mVzVPohyR80VOI0BT2uhhRws succeeded!
[SUCCESS] ASD test with DISK_ID CIkQ63CFyKWed7DsXbY9IKSMiqLemnKV succeeded!
[SUCCESS] ASD test with DISK_ID uoYCC5sBzsSj4pPnWaLPziWRFFjovo48 succeeded!
[SUCCESS] ASD test with DISK_ID Bh0Wp4v22is1as4rMvAFiwhMsdTaS8nb succeeded!
[SUCCESS] ASD test with DISK_ID DOyQspBcA31UeBAuh27Wimzpe26kKrF6 succeeded!
[SUCCESS] ASD test with DISK_ID 3LMEHBoeod4cMmVCSGOBiodGt61cLlYy succeeded!
[SUCCESS] ASD test with DISK_ID nwX6ux8UJhSevMaerBWnBnjMAdEZRCqm succeeded!
[SUCCESS] ASD test with DISK_ID VAjRAOU5Z6j5ifYQcC2YjOCk4NMnoU4n succeeded!
[WARNING] ASD test with DISK_ID JpGu4NkSqefyN0Wjzy2IPTw27pk0vOqT failed because: unknown
[WARNING] ASD test with DISK_ID 7DhygVpgHrJK0q6X9xuB9BndjcoMDxL8 failed because: unknown
[WARNING] ASD test with DISK_ID Fo6KLMCQQoFAzHF0u1bcOjFRPY6srysa failed because: unknown
[WARNING] ASD test with DISK_ID H8OcJwwlets4kSzkN1xOsLultK41W4QR failed because: unknown
[WARNING] ASD test with DISK_ID YzSMOncTbiixJ9WIDuFccfzKxfZKNxSf failed because: unknown
[WARNING] ASD test with DISK_ID IeWoGB5FGuCjwgjXcjwkaPRc8TSkkJ37 failed because: unknown
[WARNING] Alba backend sv4-hddbackend02 should be available for VPool use with 26 asds, but there are 9 defective asds: JSlAW8XvG2GDzGGZNgXMkO8cdYooKEZd, rvl7pYCsJxi0kTCgVRQKsx9zZ1w0P540, S3KkojHnGMq8mp6Rz8LtFY9mi6jGqFFg, JpGu4NkSqefyN0Wjzy2IPTw27pk0vOqT, 7DhygVpgHrJK0q6X9xuB9BndjcoMDxL8, Fo6KLMCQQoFAzHF0u1bcOjFRPY6srysa, H8OcJwwlets4kSzkN1xOsLultK41W4QR, YzSMOncTbiixJ9WIDuFccfzKxfZKNxSf, IeWoGB5FGuCjwgjXcjwkaPRc8TSkkJ37
[INFO] Checking separate ASDs for backend sv4-hddbackend01:
[SUCCESS] ASD test with DISK_ID oUVIVpt7Xck5hTTdMrECOc199lX6T7dP succeeded!
[SUCCESS] ASD test with DISK_ID EXDm9aMGsu456yCBvqvM3yi3MZBGRUWm succeeded!
[SUCCESS] ASD test with DISK_ID bIr4nGYn7joK26sLlide7I67Uuzx53R4 succeeded!
[SUCCESS] ASD test with DISK_ID MNc8m5yk1wv2WWeQ3O7YxNJdGiK9gS17 succeeded!
[SUCCESS] ASD test with DISK_ID A43W8IeXhxEYFU6sU5xGkCV8Q7jM5OH4 succeeded!
[SUCCESS] ASD test with DISK_ID syns6L4nau6TIidH39vQ38if0Ssq4Ugh succeeded!
[SUCCESS] ASD test with DISK_ID eO5nIPuL2Q1026oaPebLXkTgpqBqLLMg succeeded!
[SUCCESS] ASD test with DISK_ID 7hOPrIhLcpaOKmjauoYYMUNobM42lvgA succeeded!
[SUCCESS] ASD test with DISK_ID KKpm8QC9nuk5jgyw30kcnniftIYkKuXd succeeded!
[SUCCESS] ASD test with DISK_ID 0nE1fzBt3DAR5TB3dPekrZT9eIHVSkfE succeeded!
[SUCCESS] ASD test with DISK_ID 5Lkf1PDkgBuj3DXnEq7tDs87Vas2s5qB succeeded!
[SUCCESS] ASD test with DISK_ID g6SVUg51VXo0MklC72GJ18s8YAzag5kT succeeded!
[SUCCESS] ASD test with DISK_ID 2I5Ky2mtRWKcIA5VV1Mh2DMiQhm673BJ succeeded!
[SUCCESS] ASD test with DISK_ID HrAhMyug1VlMfn1VyDUW1R7u6oTaN2po succeeded!
[SUCCESS] ASD test with DISK_ID zFxHBj3sGSy73NZhDsj0KFDmRNewrrab succeeded!
[SUCCESS] ASD test with DISK_ID SFq8WMRQvhyMXykGY1LLTmaFryjUcRzR succeeded!
[SUCCESS] ASD test with DISK_ID zAzGE5kiL30zAYXZGzuangO11iG7PKUE succeeded!
[SUCCESS] ASD test with DISK_ID yuhM3wRicK3w501oOYzMyLjWyLPLkMU5 succeeded!
[SUCCESS] ASD test with DISK_ID wG5s2v4xKcgchFmQmLuGDD7HM0gB5xVW succeeded!
[SUCCESS] ASD test with DISK_ID Zq3G9pCjc9EY7j0dK7g1w9XLBNAtvo52 succeeded!
[WARNING] ASD test with DISK_ID tW2z3zvWpVH5abyj1stnEkv33j23kf6n failed because: nodedown
[WARNING] ASD test with DISK_ID 3FhkQEjZmc5HOR77c1JLUguZWynlbOIn failed because: nodedown
[WARNING] ASD test with DISK_ID 3t1Q5aYAIvmqQDOjjriXIBxYMgFd98SR failed because: nodedown
[WARNING] ASD test with DISK_ID StnMHJf4vq2HhESO1qgvcW63vR9ZLbJl failed because: nodedown
[SUCCESS] ASD test with DISK_ID Bd4HOyIIA53UJAU7kDYBylrzLD4QRBUd succeeded!
[SUCCESS] ASD test with DISK_ID mq7eDIz9FT9WyXeK7wLhk9MyKdmK9w9e succeeded!
[SUCCESS] ASD test with DISK_ID IrtKR3cdyLOwX8r9bFOA279fMj1SHj9i succeeded!
[SUCCESS] ASD test with DISK_ID 122KNhftcOJLxK2jnRZ0wC6JJG13u86Y succeeded!
[SUCCESS] ASD test with DISK_ID Q6SduC1xOiUQ1ZSq1nqIMrpKwA4Wbn3y succeeded!
[SUCCESS] ASD test with DISK_ID jBuVbfYHzLh4RUN4yK3EqVPK3QyYF9AR succeeded!
[SUCCESS] ASD test with DISK_ID Schqo9pHu7vezRkOBV99e6k9GyRKsc04 succeeded!
[SUCCESS] ASD test with DISK_ID BVPpG8KvJqwGj2VNSZvYbLSu0kIMCiZo succeeded!
[SUCCESS] ASD test with DISK_ID 6ugtFQSabKfy90eTCEwATfed6FsmaZsR succeeded!
[SUCCESS] ASD test with DISK_ID DNTpdrwpn8P3f2IxUnsgXDm2c5PtcMs8 succeeded!
[SUCCESS] ASD test with DISK_ID y63aqEtTDjlRsE6uolVXOuJLebtpHv11 succeeded!
[SUCCESS] ASD test with DISK_ID ty5VAkBu5odEMZhw4aoYLLzcRI3vuNXs succeeded!
[SUCCESS] ASD test with DISK_ID bNVeQa9yvwkuen9gsvZp2PRl0OoBJyEW succeeded!
[SUCCESS] ASD test with DISK_ID 1T446PGGchlLArRCepG4DuSOuKxe4FSw succeeded!
[SUCCESS] ASD test with DISK_ID Pb8R60dXKPoowovY6wWJ9Sw75sdV4tLA succeeded!
[SUCCESS] ASD test with DISK_ID f3pW4W8vMiz2AxW7nMAyuFyBIRihOQec succeeded!
[SUCCESS] ASD test with DISK_ID lDictyEmV4kUxnxGUFsuQd7568JCUj5p succeeded!
[SUCCESS] ASD test with DISK_ID VuNreXP10aSpdRKurVorpzDUYO12TJ9F succeeded!
[SUCCESS] ASD test with DISK_ID f3zfNBtdo0LLiLrBYSKYZ2IJHOHqZkpt succeeded!
[SUCCESS] ASD test with DISK_ID fSH77yWgKMzhzH4M8h2s4yBbmxIfNiUA succeeded!
[SUCCESS] ASD test with DISK_ID RCeDqL5APlCKg2etiUygVkPQnswMSL20 succeeded!
[SUCCESS] ASD test with DISK_ID aPujLp1bx7R6nK3SOjGh8Q1KKEyO21US succeeded!
[SUCCESS] ASD test with DISK_ID He2LWyGkpiCYaqJzBpxVBFdDGt5vlK0J succeeded!
[SUCCESS] ASD test with DISK_ID wtIrhuY2GbfSwTvmuGOLqiQ9LruD5q0a succeeded!
[SUCCESS] ASD test with DISK_ID SphP5xLHqeIjfHP2exWvcKhgtxOLAcLi succeeded!
[SUCCESS] ASD test with DISK_ID 6vJ6fTJkunoakyvHKlzpYw2X47Zs7Lfo succeeded!
[SUCCESS] ASD test with DISK_ID EwnL3vhRmhs4g6GexNEV6N4Un6Mm3w7A succeeded!
[SUCCESS] ASD test with DISK_ID 6hFS3rFEr4l1h9pMA7WrNchZdZaS4qs5 succeeded!
[SUCCESS] ASD test with DISK_ID F5qGOMdcH1DDxBgkCQxdQd1YLNAy8QoE succeeded!
[SUCCESS] ASD test with DISK_ID VEoza14ASGBPuHURkpqkevryIJ4IWOfj succeeded!
[SUCCESS] ASD test with DISK_ID 0ozDPSMfCCpqF8Az0NA5m709pF9Ve3c2 succeeded!
[SUCCESS] ASD test with DISK_ID bOLrYNo4Vq7OseSSMSSkGwSfKw5Tn5bX succeeded!
[SUCCESS] ASD test with DISK_ID qphQ9dyq9qxkWZ2HYxSSOo8qTxHb9b5b succeeded!
[SUCCESS] ASD test with DISK_ID NtDi5ULVbSR9XvyJBtk2HJ2MIFLwaw3d succeeded!
[SUCCESS] ASD test with DISK_ID AHoTgoXQducNqc4Xz7ayk3sKSBR0nTtZ succeeded!
[SUCCESS] ASD test with DISK_ID 0bDyMVOnJ017Sr8AKNA6LXCznH3AUvZx succeeded!
[SUCCESS] ASD test with DISK_ID fSNi6N0HHNyV53QbZUEAamGY4pBIdZid succeeded!
[SUCCESS] ASD test with DISK_ID pncz3dBPA1BQI0d0deIz8RpX73EN5LYa succeeded!
[WARNING] ASD test with DISK_ID C8O0sa7DQdgQWluSObL8TJJGudfVzOOn failed because: unknown
[WARNING] ASD test with DISK_ID R32nqOZCuJndKJPvchvDZOBW1XTJxQ3r failed because: unknown
[WARNING] ASD test with DISK_ID jsCrAIlcQsZykWzDsQDUfhjc6uDImXpS failed because: unknown
[WARNING] ASD test with DISK_ID 9x9PuLbP8zyRAT6x42k6j8Fg9kow5mdK failed because: unknown
[WARNING] ASD test with DISK_ID TDZlEEO9mrPesnPvOluR89JRH7bvQnAC failed because: unknown
[WARNING] ASD test with DISK_ID sY653FPRoieu4PSgeW8P3fSvq5SgrqQ9 failed because: unknown
[WARNING] ASD test with DISK_ID YVuzlp8UAulH6cVjFJxuzdL2jJ406sOu failed because: unknown
[WARNING] ASD test with DISK_ID nyk6rnnCyUyO50leEjCoSQXRuMW7IwWq failed because: unknown
[WARNING] Alba backend sv4-hddbackend01 should be available for VPool use with 58 asds, but there are 12 defective asds: tW2z3zvWpVH5abyj1stnEkv33j23kf6n, 3FhkQEjZmc5HOR77c1JLUguZWynlbOIn, 3t1Q5aYAIvmqQDOjjriXIBxYMgFd98SR, StnMHJf4vq2HhESO1qgvcW63vR9ZLbJl, C8O0sa7DQdgQWluSObL8TJJGudfVzOOn, R32nqOZCuJndKJPvchvDZOBW1XTJxQ3r, jsCrAIlcQsZykWzDsQDUfhjc6uDImXpS, 9x9PuLbP8zyRAT6x42k6j8Fg9kow5mdK, TDZlEEO9mrPesnPvOluR89JRH7bvQnAC, sY653FPRoieu4PSgeW8P3fSvq5SgrqQ9, YVuzlp8UAulH6cVjFJxuzdL2jJ406sOu, nyk6rnnCyUyO50leEjCoSQXRuMW7IwWq
[INFO] Checking separate ASDs for backend sv4-ssdbackend01:
[SUCCESS] ASD test with DISK_ID Ri00AwSaKhDv3lPkJ8uRad8TUhbq9iQH succeeded!
[SUCCESS] ASD test with DISK_ID VHoynWJwb0FhXuzf9tvBbDS78c2P73sO succeeded!
[SUCCESS] ASD test with DISK_ID E8oobKsfaJJhpwnr95Ud2QYRKYX1DrCy succeeded!
[SUCCESS] ASD test with DISK_ID IPWz28JwBoTvMf6e0gjocIkrU11ySisS succeeded!
[SUCCESS] ASD test with DISK_ID 1Wlu3Mrq0psaCCx1BGFKtBPtsuyCP8Ko succeeded!
[SUCCESS] ASD test with DISK_ID P5u40kvwHesaJDhFftUM5OYQV8QdZ8k9 succeeded!
[SUCCESS] ASD test with DISK_ID MSi5oNV5X0lmMeTldq181eO49jySa7Ig succeeded!
[SUCCESS] ASD test with DISK_ID Gn2vyw34EnNzwDu3QhjX0TRnhjUB6ejY succeeded!
[SUCCESS] ASD test with DISK_ID AzorLFjYcwazpFjH9e2kiBeJ7PmhbMlo succeeded!
[SUCCESS] ASD test with DISK_ID WMemhleQdvhlqupkiPCgbKDSJgTWfFT7 succeeded!
[SUCCESS] ASD test with DISK_ID 9E8rOHzYkZVW30Bv04kyUf9EJ9QS1dVz succeeded!
[SUCCESS] ASD test with DISK_ID Tuku9yExBSqV0BRNNaLNMkcQkneNLvQ8 succeeded!
[SUCCESS] ASD test with DISK_ID 91zmoIpvPHASXq5Fehut1HTcE7S6FIZL succeeded!
[SUCCESS] ASD test with DISK_ID EcWxGTzfzOBrHEHg0yPbBqjmMLtAoIlq succeeded!
[SUCCESS] ASD test with DISK_ID vJP0kXYpIb0qb8j7WVdTeRPlejh8Lryi succeeded!
[SUCCESS] ASD test with DISK_ID 5eoLARgL1ftjAATOO3Hs7RepVSmVWr56 succeeded!
[SUCCESS] ASD test with DISK_ID xQnL6Y7pFmWizhtxfYXzX1whbTjN4HN8 succeeded!
[SUCCESS] ASD test with DISK_ID eDu3Hlgj8VLmBhCAoBEUr8kBdMgfkLbO succeeded!
[SUCCESS] ASD test with DISK_ID Ycp8Yy7QBOMVaEt0JW5Rcdbr1qbxED1Y succeeded!
[SUCCESS] ASD test with DISK_ID 7OO53i2KEK8Vt5OrdIRenjucaXde3lyg succeeded!
[SUCCESS] ASD test with DISK_ID FclnxRtjQvXskaSbueqHtxgl6B2sdetl succeeded!
[SUCCESS] ASD test with DISK_ID cvvu4Rh0KMt2ToJjo93wiYMnezKrRBp5 succeeded!
[SUCCESS] ASD test with DISK_ID CvQJ8gBD19C7neuV4PtXsDedelMyqwnr succeeded!
[SUCCESS] ASD test with DISK_ID TAo90qrfM7xd62dLGxQut8xI6nh3Mkpy succeeded!
[SUCCESS] ASD test with DISK_ID 186lDrtv9ptx8P2oWoGBLAaWt5D33Yym succeeded!
[SUCCESS] ASD test with DISK_ID G6rEQX9DOmpNzsDEirhQ2kWpTyq5W81W succeeded!
[SUCCESS] ASD test with DISK_ID vxTDWaoCa7Z9EQe4Rw8ZmNMsIlnAwtmr succeeded!
[SUCCESS] ASD test with DISK_ID B9XUkwECXvtGCxXcXvorBjx2kd0loPeq succeeded!
[SUCCESS] ASD test with DISK_ID DxYokjdZXnGH9zVrWfXyMe2mVW9LwJAe succeeded!
[SUCCESS] ASD test with DISK_ID MZXhBzr6quRnDhPhFiTDOvWomAhP1EVk succeeded!
[SUCCESS] ASD test with DISK_ID RWEclgsIcijGmAP7txXI1TzPA4cTEHrw succeeded!
[SUCCESS] ASD test with DISK_ID kzxemoocRSecis4In1u355Z64F4gBWQE succeeded!
[SUCCESS] ASD test with DISK_ID Ip1rcjPf888yNLrGmtqUSS0DI8OeYFbc succeeded!
[SUCCESS] ASD test with DISK_ID 9VqB1ZGW9UpbZs8aYRME4Ei9ufmbRXza succeeded!
[SUCCESS] ASD test with DISK_ID sW5fPit16Z6DtPO6eCpY7kKWNNK8mcQU succeeded!
[SUCCESS] ASD test with DISK_ID LZI1ANavF748VTzSucF8aCkJoaIwBCIF succeeded!
[SUCCESS] ASD test with DISK_ID oXkgmS8HLoZy5uNbjvZBalVLu31LoVnG succeeded!
[SUCCESS] ASD test with DISK_ID ejALQJim51vO3QEsSmNE69vhJBiGyflO succeeded!
[SUCCESS] ASD test with DISK_ID f7dhQhuPmbM8wgctWQOwrGnJhymL23Zl succeeded!
[SUCCESS] ASD test with DISK_ID sNDoOyV6qYb4hAnAuV0dv65vjEuiKxhs succeeded!
[SUCCESS] ASD test with DISK_ID 2UXQ72C3PzXNXgTF8OOCRkUIsQdkwJyS succeeded!
[SUCCESS] ASD test with DISK_ID eWVUzU7RwYd6peJCKFRHazYTb1aqEHRF succeeded!
[SUCCESS] ASD test with DISK_ID Q32CcBP1vncOJ01r6MG5whRd5wnTk5WD succeeded!
[SUCCESS] ASD test with DISK_ID oGOnpqMiZ84RkAKpN26SdYUcI9IHNR65 succeeded!
[SUCCESS] ASD test with DISK_ID C0DH0mkncBGiROFlXM4Xk826zdjHxwR8 succeeded!
[SUCCESS] ASD test with DISK_ID mV6EjsuGGGPz6uSp1KkbzN9LWKa6imq1 succeeded!
[SUCCESS] ASD test with DISK_ID HVm33pHx68DKs6bGMOykhuGzaIyCzq5J succeeded!
[SUCCESS] ASD test with DISK_ID 9qEItHb08qcvPk2dgx357vn3Szy9CfRu succeeded!
[SUCCESS] ASD test with DISK_ID mR3t6lUgHfTZ5PJdc4P1CCBIyWWTmkRQ succeeded!
[SUCCESS] ASD test with DISK_ID LpLWaP36eap3j3rHcd5vERyFJsPqXgJO succeeded!
[SUCCESS] ASD test with DISK_ID 6RHJjOPtiyc5IdPjEEccRGWOuiBaXXOd succeeded!
[SUCCESS] ASD test with DISK_ID Ily33U8h8nygmNNkv0aZ4GZhwvoDGtok succeeded!
[SUCCESS] ASD test with DISK_ID kn36O0HGleTiAbk491x5bTLuIkiJU7Jt succeeded!
[SUCCESS] ASD test with DISK_ID CJzGUh7cvfZOk7Zew6zBjuiWJInGtD6o succeeded!
[SUCCESS] ASD test with DISK_ID iag3vGWXAg0O1U9o2BCBFDnPu2j1Lau1 succeeded!
[SUCCESS] ASD test with DISK_ID araX7RKXhOpHSzvYm9pDaoZoaO2EbqdU succeeded!
[SUCCESS] ASD test with DISK_ID 2sMSKSTqnNPnQSUfpACI69ZXGMNCBmj5 succeeded!
[SUCCESS] ASD test with DISK_ID DWXNGkI0ZmZnyG3CZWF551XrJrA9wN5G succeeded!
[SUCCESS] ASD test with DISK_ID my4NNubJVe0YLo64V2TPF8VcbxRXJfpr succeeded!
[SUCCESS] ASD test with DISK_ID Demrk5YCUUrVEiOimGpOD7fw0lDDqkZo succeeded!
[SUCCESS] ASD test with DISK_ID 02xpv9bwHOOAaex85cQz4WgO71VlYjfF succeeded!
[SUCCESS] ASD test with DISK_ID xSxZiNDvESKAAMVaWx6qJtJLSKgYeyiI succeeded!
[SUCCESS] ASD test with DISK_ID mNDPGiYqeEh9dIVrEQk2noRwJF1jRIMs succeeded!
[SUCCESS] ASD test with DISK_ID yV6PR9FHoxJBnYEjXgHBiIDUg0uq0eYz succeeded!
[SUCCESS] ASD test with DISK_ID 68Y1duTumEiGAX8zKOfxPbZJ39zlRC5Y succeeded!
[SUCCESS] ASD test with DISK_ID EcWw26QNZFDe5MxFRMaqy8Fhx0WiODIl succeeded!
[SUCCESS] ASD test with DISK_ID B80UdgvYES89sPX5SKObFg6uR7SnMjdI succeeded!
[SUCCESS] ASD test with DISK_ID 5evaYKlQZHcytJ0TsVkrCFBYpjdkcm99 succeeded!
[SUCCESS] ASD test with DISK_ID tvNjEQy29AaDOKh2GPpQTsAS1IFnj59z succeeded!
[SUCCESS] ASD test with DISK_ID IEKn4UbniK8AYogjPdQFuTEcz2VxaYfs succeeded!
[SUCCESS] ASD test with DISK_ID mWladaizsBQji4JpeNLsvg8Kf6nOfKLZ succeeded!
[SUCCESS] ASD test with DISK_ID 4BfLzOyznfd3kkwfDUAnvvkqbSd4MV7s succeeded!
[SUCCESS] ASD test with DISK_ID 5IZjpDMfWFPFjuuhq9CjtUaiEak7nxgw succeeded!
[SUCCESS] ASD test with DISK_ID Okk1JCtp4LbV8wqIi1kPXS45GaEWoGCG succeeded!
[SUCCESS] ASD test with DISK_ID vf5i0PEQyrjIN9Iszvy5y99RM1fxUsbJ succeeded!
[SUCCESS] ASD test with DISK_ID UtvYZuw4zUt9ZyGtel0NhnYRUtUKi12G succeeded!
[SUCCESS] ASD test with DISK_ID YshMo0S5EuJDySXaaYROCTYW64HggzWL succeeded!
[SUCCESS] ASD test with DISK_ID CfBN9Waq2gDff7Y1nf3m8VFVAD97oPsU succeeded!
[SUCCESS] ASD test with DISK_ID EfG1uvWpqOoY5C2Ywd9We889bw37cDpJ succeeded!
[SUCCESS] ASD test with DISK_ID odU2grCWkgiEyRQg5qsPpzo1VczwZ0x3 succeeded!
[SUCCESS] ASD test with DISK_ID DxAlMnITbsUbhCGwnkPvOIpWrtuDzpaT succeeded!
[SUCCESS] Alba backend sv4-ssdbackend01 should be available for VPool use. All asds are working fine!
[INFO] Checking separate ASDs for backend sv4-ssdbackend02:
[SUCCESS] ASD test with DISK_ID bahkcd6tMo5QrpF9WQJwHpcalfQdG20r succeeded!
[SUCCESS] ASD test with DISK_ID klcXtlJDbQn32g8GwfNzffRWpQCWpmAZ succeeded!
[SUCCESS] ASD test with DISK_ID ifwovUzyq0GGz2y9C26fzFUQpsvSQeYd succeeded!
[SUCCESS] ASD test with DISK_ID fmYAosgs1KAvsm3IZxVVeW8fpMWEt0GC succeeded!
[SUCCESS] ASD test with DISK_ID LLdImGr5xDGbAYyGl6TkBmFuWVcQyYxG succeeded!
[SUCCESS] ASD test with DISK_ID S2fFelPj5RJ2Vz64kkkfSc5qVuoKAmYz succeeded!
[SUCCESS] ASD test with DISK_ID vtg2xTSyYex7GjCUWWAKmqmj43dvlPYs succeeded!
[SUCCESS] ASD test with DISK_ID vwiPLcKqO4CKDxOglAef2HOkwZezRqqn succeeded!
[SUCCESS] ASD test with DISK_ID NYvZNQdnGTVIUIx41a0d41o96RMz5CSn succeeded!
[SUCCESS] ASD test with DISK_ID FofTlu4eRGlg0ANAxOWpr9jvlYLHKwnH succeeded!
[SUCCESS] ASD test with DISK_ID OLrgCDqmFjaC0SjvPK96zdI8fibC6DGE succeeded!
[SUCCESS] ASD test with DISK_ID w742vl9UMhtWPZ6xK4DEUEGu3zxfHCQM succeeded!
[SUCCESS] ASD test with DISK_ID Me2EzOLniVzqmhsufJgZ0P8TCDIWCcyt succeeded!
[SUCCESS] ASD test with DISK_ID 5K5KvJgEqoHVub0buwgdpxAlz8FKXL0O succeeded!
[SUCCESS] ASD test with DISK_ID KRzIEW8ZJmGwEDuLaitopTZIk0oDCWZ9 succeeded!
[SUCCESS] ASD test with DISK_ID t7HJgCUcd3j8j91qt1eBHVYotZXf4cw1 succeeded!
[SUCCESS] ASD test with DISK_ID tLISxfSxqEvaBckLFIxBouNGF76pkdkM succeeded!
[SUCCESS] ASD test with DISK_ID CVUfP3iPW5a4WMCrp0KjYNqbAk54Y5EV succeeded!
[SUCCESS] ASD test with DISK_ID nVBkTtzHXgw3cabH1YUztZBlKEtQsrwZ succeeded!
[SUCCESS] ASD test with DISK_ID 0tX8hHuOzCGUXKbnZlYofRGnfQLc6kBC succeeded!
[SUCCESS] ASD test with DISK_ID aBmWhaKBAplJ66ORRQoJEHFIKbXrQeMe succeeded!
[SUCCESS] ASD test with DISK_ID cxcPp8n9s0N4QWowPe3eMPCJhkw5rklL succeeded!
[SUCCESS] ASD test with DISK_ID gaPsJUcSPRGLjiGsGBMgLnYCusxmoy8z succeeded!
[SUCCESS] ASD test with DISK_ID zJTvxnQHvHURHYOcYHHMTtrIvMWgALSj succeeded!
[SUCCESS] ASD test with DISK_ID xswlfpgBznVNbEWtf6RyKV3WNXxPblPr succeeded!
[SUCCESS] ASD test with DISK_ID iopIMVP5hOsg8FpJHXPF301VKzTyp7Og succeeded!
[SUCCESS] ASD test with DISK_ID I44WArlkPmE3hsVyMnky0geXwgGbeFnf succeeded!
[SUCCESS] ASD test with DISK_ID xZeMhIYtKUPf9QAA8452SR9rvXT8ijhp succeeded!
[SUCCESS] ASD test with DISK_ID 6LsazgDXFnA4envKOz1y2957kQKURnE1 succeeded!
[SUCCESS] ASD test with DISK_ID d0fEeEoFrOGHdIjkIRSjmufUJLvlx9CM succeeded!
[SUCCESS] ASD test with DISK_ID E64s9mvjxnACC8M7Y70CulVWgQg3MXVK succeeded!
[SUCCESS] ASD test with DISK_ID Yeo9RkNULMrmmXEoEgR2dRVQk1B4ixUI succeeded!
[SUCCESS] ASD test with DISK_ID sVnRBHX9FqwOGbRZtk3XLgUjRZk2sWuL succeeded!
[SUCCESS] ASD test with DISK_ID rx6mQSBWvHDCIG3Ko0qGxqCL0EIU2hIe succeeded!
[SUCCESS] ASD test with DISK_ID koKhSG954pPMMacvNJm5PokccgvR8gK3 succeeded!
[SUCCESS] ASD test with DISK_ID QTIW0yJMplUSflk0J2NKkTgXm1fQSJoQ succeeded!
[SUCCESS] ASD test with DISK_ID tCMDoBXkYAG3aEpAGFzOdWXk8XxqeJ4f succeeded!
[SUCCESS] ASD test with DISK_ID hYgj6sQdB5ZccXXakbzuYTSbjXb1mARY succeeded!
[SUCCESS] ASD test with DISK_ID hN9FaHJ1OgepUblFuaEzI0yxhtS0wu7f succeeded!
[SUCCESS] ASD test with DISK_ID GavEWQM4ajFtG9qkBGPAx5wX1bp8qrbx succeeded!
[SUCCESS] ASD test with DISK_ID orRidPXAP6SrlcKSY144QYvAUEbC5iaZ succeeded!
[SUCCESS] ASD test with DISK_ID qqy2q5AlvDhEkUcsTQd3U3Rhn316dbPJ succeeded!
[SUCCESS] ASD test with DISK_ID 7gdM9BZFNOZtj1Y4ajn2SvGE8AIPucEg succeeded!
[SUCCESS] ASD test with DISK_ID 9asoK86MdPxAZ0aUasFstxAPSOOuCnmo succeeded!
[SUCCESS] ASD test with DISK_ID Xc2SeVVxf7DDcbR4EhFLt9mnO5MCGwSG succeeded!
[SUCCESS] ASD test with DISK_ID 2Bs5QZEyCHfbq1qk6LVQ5WpKkiS08q0E succeeded!
[SUCCESS] ASD test with DISK_ID DAxORxGdzFDuikHZ5b7oh04cZrxqjRLY succeeded!
[SUCCESS] ASD test with DISK_ID aXfvbyrtfsKBomCA5qIKcwiIPALHEpqs succeeded!
[SUCCESS] ASD test with DISK_ID Z9wTIziaTAJBkt0CWBcc5KjPnK7TZovR succeeded!
[SUCCESS] ASD test with DISK_ID rZ2mrxtwbytUK2dah96pRSbzqP5qtcT3 succeeded!
[SUCCESS] ASD test with DISK_ID uhchCxdUJ7Yj4xLAH5RESvH2fmSmBM5z succeeded!
[SUCCESS] ASD test with DISK_ID B30z87Q4TXyPDIyrFtGPY8W7Q3nywkws succeeded!
[SUCCESS] ASD test with DISK_ID RMSw1L3bUvsn3OzuKoL4DquEqDw9zten succeeded!
[SUCCESS] ASD test with DISK_ID yVh2a323THyRe2fLAPpKEcjCqfGFl5ax succeeded!
[SUCCESS] ASD test with DISK_ID Af5EWvxoLJ873udllOLrLxb7dTY8ktKB succeeded!
[SUCCESS] ASD test with DISK_ID veyfsfpTRgdTVvuQGy6RDHE7qtGBGZf3 succeeded!
[SUCCESS] ASD test with DISK_ID 9cD3JnKaHFfg83OFSUrHCocwFuCHlRp4 succeeded!
[SUCCESS] ASD test with DISK_ID 3Ej54cLMsmy9Qk6Or1gNPDKtmDR7fsvd succeeded!
[SUCCESS] ASD test with DISK_ID VDCvIpL9oRTLtPJMlCmnLBTldakHC2dy succeeded!
[SUCCESS] ASD test with DISK_ID 5zIdYLaKRJWSEdKszSbW9LOmLyTqZcV0 succeeded!
[SUCCESS] ASD test with DISK_ID pubF0BZC9wqEKCMGc5tNMUs40PzBMb3b succeeded!
[SUCCESS] ASD test with DISK_ID gcwEAHaofkpctBVXrBhX49mn43GErpiE succeeded!
[SUCCESS] ASD test with DISK_ID UaDs27COHtDQlIWj6KfndPgf5E1kEJl6 succeeded!
[SUCCESS] ASD test with DISK_ID xWeW2BnumZ7YEmtsjqN95PGueV5ItmCl succeeded!
[SUCCESS] ASD test with DISK_ID VWSiIhBJl0w5tUlYMEOH2LG9iWPQqaZN succeeded!
[SUCCESS] ASD test with DISK_ID fsZqTf6ELNBOOLuTPOV8edpPdkz0Ejw6 succeeded!
[SUCCESS] ASD test with DISK_ID LKiAZD9XE6nItxIuDPjQWM2P3TvUgQzt succeeded!
[SUCCESS] ASD test with DISK_ID vEOtn1EbWItv7zWzcan1AfmB0gbbpYgC succeeded!
[SUCCESS] ASD test with DISK_ID 7UjIWyEYR73Vjxz3XC1GoYURQz1IhYPJ succeeded!
[SUCCESS] ASD test with DISK_ID ptjP6e51FnIVWaQPGi1sjWdUnpaMyXkP succeeded!
[SUCCESS] ASD test with DISK_ID tRxSgNiLS2hBxurOthgSOBMTbxoahari succeeded!
[SUCCESS] ASD test with DISK_ID 4EWM8coeLx1hpZK7Uo4SbK6gNW1o9v4x succeeded!
[SUCCESS] ASD test with DISK_ID WTcLPFttmd46hVeUehJNa8Oo4I1b3gL8 succeeded!
[SUCCESS] ASD test with DISK_ID 3ecmEDwcrXDEaxoL6dsv1U3bXyXhZiXp succeeded!
[SUCCESS] ASD test with DISK_ID 7WkxO7GoqudvwxdXWTstk18WsI3HhuuW succeeded!
[SUCCESS] ASD test with DISK_ID mubxaRQFsO84Vr6wBo7aVj2P1nzhLFUF succeeded!
[SUCCESS] ASD test with DISK_ID dUpuzurYX9sQ5NvZVZRWmQGbOZ3ICt2S succeeded!
[SUCCESS] ASD test with DISK_ID 7qcrAmUf9iHrKGzQpNt0yDnAhuJpRRAM succeeded!
[SUCCESS] ASD test with DISK_ID Nyd0SgbiYIxez92rKhxOhOfWmMaW577T succeeded!
[SUCCESS] Alba backend sv4-ssdbackend02 should be available for VPool use. All asds are working fine!
[INFO] Recap of Health Check module alba test backend-test!
[INFO] ======================
[INFO] SUCCESS=272 FAILED=0 SKIPPED=0 WARNING=33 EXCEPTION=0

Copied from original issue: openvstorage/openvstorage-health-check#434

openvstorage-ci commented 6 years ago

From @JeffreyDevloo on February 13, 2018 9:39

Looks like an issue of the Framework Alba Plugin instead. Fetching disk information was not possible.

matthiasdeblock commented 6 years ago

Had the same issue on the SV4 cluster. Restarting the asd-manager on the nodes which showed red ASD's temporary fixed the problem.

Error from the asd-manager:

2018-07-07 03:54:42 29700 -0400 - SV4SRV0065 - 10344/140292296378112 - asd-manager/api - 279164 - ERROR - Unexpected exception
Traceback (most recent call last):
  File "/opt/asd-manager/source/app/decorators.py", line 77, in new_function
    return_data = f(*args, **kwargs)
  File "/opt/asd-manager/source/app/api.py", line 94, in list_disks
    DiskController.sync_disks()
  File "/opt/asd-manager/source/controllers/disk.py", line 57, in sync_disks
    for path_type in DiskController._local_client.dir_list(directory='/dev/disk'):
  File "/opt/asd-manager/source/tools/localclient.py", line 266, in dir_list
    return os.listdir(directory)
OSError: [Errno 24] Too many open files: '/dev/disk'
JeffreyDevloo commented 5 years ago

We might be leaking file descriptors in the asd-manager. Could you provide the lsof information if it occurs again?