search

threefoldtecharchive / 0-robot

Distributed live cycle management system
Apache License 2.0
0 stars 0 forks source link

restarting zrobot container in kds farm produces many errors #49

Closed 0xIslamTaha closed 5 years ago

0xIslamTaha commented 5 years ago

Description I restarted the 0-robot containers in kds farm using zrobot_continer.stop() method, After a while all zrobots have been back to work but I can't create any new service and all nodes are down in the grid.

I checked the zrobot logs and here is it 

In [85]: job = zrobot.client.subscribe('zrobot', 'zrobot.logs')

In [86]: job.stream()
[Thu15 11:49] - base.py           :214 :29ecdf2a-5a48-482d-bf13-1829c42bdbaf-176be164-492d-424c-8117-b4078a0f5389 (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :836b5c8f-6849-4cca-970b-250028e13c81-946c0257-2d29-4ca6-a16e-69a75054c3c9 (zerodb) - INFO     - Monitor zerodb 836b5c8f-6849-4cca-970b-250028e13c81
[Thu15 11:49] - base.py           :214 :836b5c8f-6849-4cca-970b-250028e13c81-946c0257-2d29-4ca6-a16e-69a75054c3c9 (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :28b43244-670d-4847-84ba-13eadecf1b6f-22a1d405-4cfb-48ca-9bca-69b78f8cdffd (zerodb) - INFO     - Monitor zerodb 28b43244-670d-4847-84ba-13eadecf1b6f
[Thu15 11:49] - base.py           :214 :28b43244-670d-4847-84ba-13eadecf1b6f-22a1d405-4cfb-48ca-9bca-69b78f8cdffd (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :baada119-0717-4e8c-aab3-b0504738297a-b592050b-fa1f-4ff9-bc4a-6b113ec7f9fa (zerodb) - INFO     - Monitor zerodb baada119-0717-4e8c-aab3-b0504738297a
[Thu15 11:49] - base.py           :214 :baada119-0717-4e8c-aab3-b0504738297a-b592050b-fa1f-4ff9-bc4a-6b113ec7f9fa (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :e82c91f1-411a-49dc-88d7-dd37e76d4b2b-f105d9ce-8d1c-4d8e-b748-d19fcf0aedbb (zerodb) - INFO     - Monitor zerodb e82c91f1-411a-49dc-88d7-dd37e76d4b2b
[Thu15 11:49] - base.py           :214 :e82c91f1-411a-49dc-88d7-dd37e76d4b2b-f105d9ce-8d1c-4d8e-b748-d19fcf0aedbb (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - base.py           :214 :84e872e3-0b07-4b72-a9a7-90c89490deae_s3vm-535eea0e-6fa0-49c5-8bb7-b846d3b383f9 (vdisk) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 58, in get
    raise StateCategoryNotExistsError("category %s does not exist" % category)
zerorobot.template.state.StateCategoryNotExistsError: category actions does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/vdisk/vdisk.py", line 46, in _monitor
    self.state.check('actions', 'install', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state actions:install:ok failed

[Thu15 11:49] - zerodb.py         :35  :5a0e2ec0-d7dc-412d-abb3-a38caf811042-fb9725b8-3516-4411-a02a-ad6769a481ec (zerodb) - INFO     - Monitor zerodb 5a0e2ec0-d7dc-412d-abb3-a38caf811042
[Thu15 11:49] - base.py           :214 :5a0e2ec0-d7dc-412d-abb3-a38caf811042-fb9725b8-3516-4411-a02a-ad6769a481ec (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :a4d9fb98-7144-4f59-b29c-572ffb657ebf-feda7601-9119-4fb7-a422-eb075017d372 (zerodb) - INFO     - Monitor zerodb a4d9fb98-7144-4f59-b29c-572ffb657ebf
[Thu15 11:49] - base.py           :214 :a4d9fb98-7144-4f59-b29c-572ffb657ebf-feda7601-9119-4fb7-a422-eb075017d372 (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :91297fc6-2904-4b51-8859-2981feccb0fb-24099998-a34b-4017-b6a5-2bc8b56d47fd (zerodb) - INFO     - Monitor zerodb 91297fc6-2904-4b51-8859-2981feccb0fb
[Thu15 11:49] - base.py           :214 :91297fc6-2904-4b51-8859-2981feccb0fb-24099998-a34b-4017-b6a5-2bc8b56d47fd (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :2ae0f3aa-9acf-4e34-bfb4-ee6394e61a9b-f273c8ad-8c77-4731-89c0-f384c4dc88b2 (zerodb) - INFO     - Monitor zerodb 2ae0f3aa-9acf-4e34-bfb4-ee6394e61a9b
[Thu15 11:49] - base.py           :214 :2ae0f3aa-9acf-4e34-bfb4-ee6394e61a9b-f273c8ad-8c77-4731-89c0-f384c4dc88b2 (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :58fb6c72-f6e3-44cd-b0fe-65ea0a2df078-29ded04a-88f1-474f-b605-9f90d63a65f2 (zerodb) - INFO     - Monitor zerodb 58fb6c72-f6e3-44cd-b0fe-65ea0a2df078
[Thu15 11:49] - base.py           :214 :58fb6c72-f6e3-44cd-b0fe-65ea0a2df078-29ded04a-88f1-474f-b605-9f90d63a65f2 (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - vm.py             :51  :ecf434ce-5899-4b35-bc8b-6423f0b9846a_vm-1b17b98a-ca4c-4dba-bb6a-bfb1a47b2286 (vm) - INFO     - Monitor vm ecf434ce-5899-4b35-bc8b-6423f0b9846a_vm
[Thu15 11:49] - base.py           :214 :e82c91f1-411a-49dc-88d7-dd37e76d4b2b-f105d9ce-8d1c-4d8e-b748-d19fcf0aedbb (zerodb) - ERROR    - error executing action namespace_private_url:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 76, in execute
    self._result = self._func(**self._args)
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 169, in namespace_private_url
    return self._zerodb_sal.namespaces[name].private_url
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/sal_zos/zerodb/namespace.py", line 74, in private_url
    url = 'zdb://{}:9900?size={}G&blocksize=4096&namespace={}'.format(self.parent.container.default_ip().ip, self.size, self.name)
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/sal_zos/abstracts.py", line 344, in container
    self._container = self.node.containers.get(self._container_name)
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/sal_zos/container/Container.py", line 26, in get
    containers = list(self.node.client.container.find(name).values())
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/clients/zero_os_protocol/ContainerManager.py", line 305, in find
    return self._client.json('corex.find', {'tags': tags})
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/clients/zero_os_protocol/BaseClient.py", line 125, in json
    result = self.sync(command, arguments, tags=tags, id=id)
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/clients/zero_os_protocol/BaseClient.py", line 109, in sync
    response = self.raw(command, arguments, tags=tags, id=id)
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/clients/zero_os_protocol/Client.py", line 261, in raw
    self._redis.rpush('core:default', json.dumps(payload))
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 1609, in rpush
    return self.execute_command('RPUSH', name, *values)
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 754, in execute_command
    connection.send_command(*args)
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 619, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 658, in pack_command
    buffer_cutoff = self._buffer_cutoff
AttributeError: 'UnixDomainSocketConnection' object has no attribute '_buffer_cutoff'

[Thu15 11:49] - node.py           :50  :local-c9100797-895f-4c3d-b354-f3f71ac1c42b (node) - INFO     - Monitoring node local
[Thu15 11:49] - base.py           :214 :local-c9100797-895f-4c3d-b354-f3f71ac1c42b (node) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/node/node.py", line 54, in _monitor
    sp = self._node_sal.storagepools.get('zos-cache')
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/sal_zos/storage/StoragePool.py", line 66, in get
    for pool in self.list():
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/sal_zos/storage/StoragePool.py", line 56, in list
    btrfs_list = self.client.btrfs.list()
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/clients/zero_os_protocol/BtrfsManager.py", line 42, in list
    return self._client.json('btrfs.list', {})
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/clients/zero_os_protocol/BaseClient.py", line 125, in json
    result = self.sync(command, arguments, tags=tags, id=id)
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/clients/zero_os_protocol/BaseClient.py", line 109, in sync
    response = self.raw(command, arguments, tags=tags, id=id)
  File "/opt/code/github/threefoldtech/jumpscale_lib/JumpscaleLib/clients/zero_os_protocol/Client.py", line 261, in raw
    self._redis.rpush('core:default', json.dumps(payload))
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 1609, in rpush
    return self.execute_command('RPUSH', name, *values)
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 754, in execute_command
    connection.send_command(*args)
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 619, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 658, in pack_command
    buffer_cutoff = self._buffer_cutoff
AttributeError: 'UnixDomainSocketConnection' object has no attribute '_buffer_cutoff'

[Thu15 11:49] - node_port_manager.py:27  :_port_manager-0a0473ca-13cf-4d7d-ba4f-fea5e8ee1b3a (node_port_manager) - INFO     - port manager: start clean up reserved ports
[Thu15 11:49] - namespace.py      :35  :8b543cb0-c0b1-4b9d-a441-fc7f9168bf46-8b543cb0-c0b1-4b9d-a441-fc7f9168bf46 (namespace) - INFO     - Monitor namespace 8b543cb0-c0b1-4b9d-a441-fc7f9168bf46
[Thu15 11:49] - namespace.py      :35  :e08e7a29-ae70-4567-ac75-2c0b18a19b7d-e08e7a29-ae70-4567-ac75-2c0b18a19b7d (namespace) - INFO     - Monitor namespace e08e7a29-ae70-4567-ac75-2c0b18a19b7d
[Thu15 11:49] - namespace.py      :35  :27949447-aaf4-488a-8451-674a790bd4b3-27949447-aaf4-488a-8451-674a790bd4b3 (namespace) - INFO     - Monitor namespace 27949447-aaf4-488a-8451-674a790bd4b3
[Thu15 11:49] - namespace.py      :35  :a3d85152-8b10-44d3-a5fd-7a2e02a84f06-a3d85152-8b10-44d3-a5fd-7a2e02a84f06 (namespace) - INFO     - Monitor namespace a3d85152-8b10-44d3-a5fd-7a2e02a84f06
[Thu15 11:49] - namespace.py      :35  :2dc0c82c-16e7-4476-bf65-a88d94271d5f-2dc0c82c-16e7-4476-bf65-a88d94271d5f (namespace) - INFO     - Monitor namespace 2dc0c82c-16e7-4476-bf65-a88d94271d5f
[Thu15 11:49] - namespace.py      :35  :862cd41c-e60a-4079-a6fb-67f387157e4f-862cd41c-e60a-4079-a6fb-67f387157e4f (namespace) - INFO     - Monitor namespace 862cd41c-e60a-4079-a6fb-67f387157e4f
[Thu15 11:49] - zerodb.py         :35  :29ecdf2a-5a48-482d-bf13-1829c42bdbaf-176be164-492d-424c-8117-b4078a0f5389 (zerodb) - INFO     - Monitor zerodb 29ecdf2a-5a48-482d-bf13-1829c42bdbaf
[Thu15 11:49] - base.py           :214 :29ecdf2a-5a48-482d-bf13-1829c42bdbaf-176be164-492d-424c-8117-b4078a0f5389 (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :836b5c8f-6849-4cca-970b-250028e13c81-946c0257-2d29-4ca6-a16e-69a75054c3c9 (zerodb) - INFO     - Monitor zerodb 836b5c8f-6849-4cca-970b-250028e13c81
[Thu15 11:49] - base.py           :214 :836b5c8f-6849-4cca-970b-250028e13c81-946c0257-2d29-4ca6-a16e-69a75054c3c9 (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :28b43244-670d-4847-84ba-13eadecf1b6f-22a1d405-4cfb-48ca-9bca-69b78f8cdffd (zerodb) - INFO     - Monitor zerodb 28b43244-670d-4847-84ba-13eadecf1b6f
[Thu15 11:49] - base.py           :214 :28b43244-670d-4847-84ba-13eadecf1b6f-22a1d405-4cfb-48ca-9bca-69b78f8cdffd (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :baada119-0717-4e8c-aab3-b0504738297a-b592050b-fa1f-4ff9-bc4a-6b113ec7f9fa (zerodb) - INFO     - Monitor zerodb baada119-0717-4e8c-aab3-b0504738297a
[Thu15 11:49] - base.py           :214 :baada119-0717-4e8c-aab3-b0504738297a-b592050b-fa1f-4ff9-bc4a-6b113ec7f9fa (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :e82c91f1-411a-49dc-88d7-dd37e76d4b2b-f105d9ce-8d1c-4d8e-b748-d19fcf0aedbb (zerodb) - INFO     - Monitor zerodb e82c91f1-411a-49dc-88d7-dd37e76d4b2b
[Thu15 11:49] - base.py           :214 :e82c91f1-411a-49dc-88d7-dd37e76d4b2b-f105d9ce-8d1c-4d8e-b748-d19fcf0aedbb (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - base.py           :214 :ecf434ce-5899-4b35-bc8b-6423f0b9846a_s3vm-fdc729a6-fa7e-4878-a75b-d7397b94d036 (vdisk) - ERROR    - error executing action private_url:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/vdisk/vdisk.py", line 112, in private_url
    return self._zerodb.schedule_action('namespace_private_url', args={'name': self.data['nsName']}).wait(die=True).result
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 131, in wait
    raise RuntimeError(self.eco.message)
RuntimeError: 'UnixDomainSocketConnection' object has no attribute '_buffer_cutoff'

[Thu15 11:49] - base.py           :214 :ecf434ce-5899-4b35-bc8b-6423f0b9846a_vm-1b17b98a-ca4c-4dba-bb6a-bfb1a47b2286 (vm) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/vm/vm.py", line 69, in _monitor
    self._update_vdisk_url()
  File "/opt/code/github/threefoldtech/0-templates/templates/vm/vm.py", line 100, in _update_vdisk_url
    disk['url'] = vdisk.schedule_action('private_url').wait(die=True).result
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 131, in wait
    raise RuntimeError(self.eco.message)
RuntimeError: 'UnixDomainSocketConnection' object has no attribute '_buffer_cutoff'

[Thu15 11:49] - base.py           :214 :84e872e3-0b07-4b72-a9a7-90c89490deae_s3vm-535eea0e-6fa0-49c5-8bb7-b846d3b383f9 (vdisk) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 58, in get
    raise StateCategoryNotExistsError("category %s does not exist" % category)
zerorobot.template.state.StateCategoryNotExistsError: category actions does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/vdisk/vdisk.py", line 46, in _monitor
    self.state.check('actions', 'install', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state actions:install:ok failed

[Thu15 11:49] - zerodb.py         :35  :5a0e2ec0-d7dc-412d-abb3-a38caf811042-fb9725b8-3516-4411-a02a-ad6769a481ec (zerodb) - INFO     - Monitor zerodb 5a0e2ec0-d7dc-412d-abb3-a38caf811042
[Thu15 11:49] - base.py           :214 :5a0e2ec0-d7dc-412d-abb3-a38caf811042-fb9725b8-3516-4411-a02a-ad6769a481ec (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed

[Thu15 11:49] - zerodb.py         :35  :a4d9fb98-7144-4f59-b29c-572ffb657ebf-feda7601-9119-4fb7-a422-eb075017d372 (zerodb) - INFO     - Monitor zerodb a4d9fb98-7144-4f59-b29c-572ffb657ebf
[Thu15 11:49] - base.py           :214 :a4d9fb98-7144-4f59-b29c-572ffb657ebf-feda7601-9119-4fb7-a422-eb075017d372 (zerodb) - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 79, in check
    state = self.get(category, tag)
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 65, in get
    raise StateCategoryNotExistsError("tag %s does not exist in category %s" % (tag, category))
zerorobot.template.state.StateCategoryNotExistsError: tag mounted does not exist in category disks

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute
    self._result = self._func()
  File "/opt/code/github/threefoldtech/0-templates/templates/zerodb/zerodb.py", line 43, in _monitor
    node.state.check('disks', 'mounted', 'ok')
  File "/opt/code/github/threefoldtech/0-robot/zerorobot/template/state.py", line 86, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state disks:mounted:ok failed
0xIslamTaha commented 5 years ago

Root cause: Redis-py has been updated with breaking changes, once @maxux re-build the flists using the old version, robots r working without any issue.