s3_red disk failure , self healing not working as expected

[Pishoy]

• i have create s3_red
• corrupted the disk use echo 1 > /sys/block/sdf1/device/delete
• issue is that active/passive does not switched as old ones
• active mino takes a time to be unreachable then it started normaly
• another issue is that no replication between active and passive (When write on active not come on passive)

-- here is the steps that are done 
In [6]: robot = j.clients.zrobot.robots.get('mylocal')

In [7]: zt = robot.services.create('github.com/threefoldtech/0-templates/zerotier_client/0.0.1', 'main', args)

In [8]: data = {
   ...:     'farmerIyoOrg': 'bancadati.threefold.grid.farm01',
   ...:     'mgmtNic': {'id':'d5e5fb6537a04d41', 'ztClient': 'main'},
   ...:     'storageType': 'hdd',
   ...:     'dataShards':4, 
   ...:     'parityShards': 2,
   ...:     'storageSize': 200,
   ...:     'minioLogin': 'login',
   ...:     'minioPassword': 'password',
   ...: }

In [9]: s3_redundant = robot.services.create('github.com/threefoldtech/0-templates/s3_redundant/0.0.1','tlogs_s3_redB', data)

In [10]: s3_redundant.schedule_action('install').wait(die=True)
Out[10]: install

In [11]: _.result

In [12]: s3_redundant.schedule_action('urls').wait(die=True).result
Out[12]: 
{'active_urls': {'public': 'http://172.28.29.216:1024',
  'storage': 'http://172.16.46.18:1026'},
 'passive_urls': {'public': 'http://172.28.113.153:1024',
  'storage': 'http://172.16.46.18:1029'}}
  In [14]: s3_redundant = robot.services.get(name='tlogs_s3_redB')

In [15]: s3_redundant.data
Out[15]: 
{'data': {'activeS3': 'dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e',
  'dataShards': 4,
  'farmerIyoOrg': 'bancadati.threefold.grid.farm01',
  'mgmtNic': {'id': 'd5e5fb6537a04d41', 'ztClient': 'main'},
  'minioBlockSize': 1048576,
  'minioLogin': 'login',
  'minioPassword': 'password',
  'nsPassword': 'kz190wh0m3k1fw6dv3uz6lxe728uia5l',
  'parityShards': 2,
  'passiveS3': 'ca44551b-bcbc-4976-a352-afd2aa5118e1',
  'reverseProxy': '',
  'storageSize': 200,
  'storageType': 'hdd'},
 'guid': '03abc5b6-5f08-4470-96f8-a70487a0b5ad',
 'name': 'tlogs_s3_redB',
 'public': False,
 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'}],
 'template': 'github.com/threefoldtech/0-templates/s3_redundant/0.0.1',
 'version': '0.0.1'}

In [16]: active = robot.services.get(name='dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e',template_name='s3')

In [17]: active.data
Out[17]: 
{'data': {'activeS3': '',
  'current_namespaces_connections': ['172.16.46.76:1033',
   '172.30.111.183:1024',
   '172.30.115.224:1026',
   '172.30.177.1:1025',
   '172.30.177.1:1026',
   '172.30.196.67:1033',
   '172.30.51.94:1026',
   '172.30.98.171:1028'],
  'dataShards': 4,
  'deletableNamespaces': [],
  'farmerIyoOrg': 'bancadati.threefold.grid.farm01',
  'master': {'address': '', 'name': '', 'node': '', 'url': ''},
  'mgmtNic': {'id': 'd5e5fb6537a04d41', 'ztClient': 'main'},
  'minioBlockSize': 1048576,
  'minioLogin': 'login',
  'minioPassword': 'password',
  'minioUrls': {'public': 'http://172.28.29.216:1024',
   'storage': 'http://172.16.46.18:1026'},
  'namespaces': [{'name': '2b89be22-52ae-4339-ac12-408bd78f42f1',
    'node': 'ac1f6b479ca8',
    'url': 'http://172.30.196.67:6600'},
   {'name': '99eb2559-c30d-4a99-81c7-c5ed053dc11b',
    'node': 'ac1f6b457bd8',
    'url': 'http://172.30.115.224:6600'},
   {'name': '7486dfb0-f2ca-43b5-b7e5-14b5c611b310',
    'node': 'ac1f6b47a14c',
    'url': 'http://172.30.98.171:6600'},
   {'name': '05600722-6de2-4e13-8aae-663fe263f5f8',
    'node': 'ac1f6b457c5c',
    'url': 'http://172.30.189.204:6600'},
   {'name': '3d496174-e4da-4ff6-a615-9f3407db5337',
    'node': 'ac1f6b457d68',
    'url': 'http://172.30.51.94:6600'},
   {'name': 'e1a92d30-3027-452c-b8d2-57578eadd5e8',
    'node': 'ac1f6b457d18',
    'url': 'http://172.30.177.1:6600'},
   {'name': 'a27193b4-a680-45cf-ad41-7b5de98b9085',
    'node': 'ac1f6b479e74',
    'url': 'http://172.30.111.183:6600'},
   {'name': '156c3875-7275-46ad-aa9b-aea1b3ee8b85',
    'node': 'ac1f6b457d18',
    'url': 'http://172.30.177.1:6600'}],
  'nsName': '03abc5b6-5f08-4470-96f8-a70487a0b5ad',
  'nsPassword': 'kz190wh0m3k1fw6dv3uz6lxe728uia5l',
  'parityShards': 2,
  'passiveS3': '',
  'reverseProxy': '',
  'storageSize': 200,
  'storageType': 'hdd',
  'tlog': {'address': '172.16.46.18:1030',
   'name': 'ee072dbe-351d-4551-9095-7ab894f52ac7',
   'node': 'ac1f6b457b2c',
   'url': 'http://172.30.43.86:6600'}},
 'guid': 'dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e',
 'name': 'dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e',
 'public': False,
 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'},
  {'category': 'status', 'state': 'ok', 'tag': 'running'},
  {'category': 'vm', 'state': 'ok', 'tag': 'running'}],
 'template': 'github.com/threefoldtech/0-templates/s3/0.0.1',
 'version': '0.0.1'}

In [18]: passive = robot.services.get(name='ca44551b-bcbc-4976-a352-afd2aa5118e1',template_name='s3')

In [19]: passive.data
Out[19]: 
{'data': {'activeS3': '',
  'current_namespaces_connections': ['172.16.46.76:1033',
   '172.30.111.183:1024',
   '172.30.115.224:1026',
   '172.30.177.1:1025',
   '172.30.177.1:1026',
   '172.30.196.67:1033',
   '172.30.51.94:1026',
   '172.30.98.171:1028'],
  'dataShards': 4,
  'deletableNamespaces': [],
  'farmerIyoOrg': 'bancadati.threefold.grid.farm01',
  'master': {'address': '172.16.46.18:1030',
   'name': 'ee072dbe-351d-4551-9095-7ab894f52ac7',
   'node': 'ac1f6b457b2c',
   'url': 'http://172.30.43.86:6600'},
  'mgmtNic': {'id': 'd5e5fb6537a04d41', 'ztClient': 'main'},
  'minioBlockSize': 1048576,
  'minioLogin': 'login',
  'minioPassword': 'password',
  'minioUrls': {'public': 'http://172.28.113.153:1024',
   'storage': 'http://172.16.46.18:1029'},
  'namespaces': [{'name': '2b89be22-52ae-4339-ac12-408bd78f42f1',
    'node': 'ac1f6b479ca8',
    'url': 'http://172.30.196.67:6600'},
   {'name': '99eb2559-c30d-4a99-81c7-c5ed053dc11b',
    'node': 'ac1f6b457bd8',
    'url': 'http://172.30.115.224:6600'},
   {'name': '7486dfb0-f2ca-43b5-b7e5-14b5c611b310',
    'node': 'ac1f6b47a14c',
    'url': 'http://172.30.98.171:6600'},
   {'name': '05600722-6de2-4e13-8aae-663fe263f5f8',
    'node': 'ac1f6b457c5c',
    'url': 'http://172.30.189.204:6600'},
   {'name': '3d496174-e4da-4ff6-a615-9f3407db5337',
    'node': 'ac1f6b457d68',
    'url': 'http://172.30.51.94:6600'},
   {'name': 'e1a92d30-3027-452c-b8d2-57578eadd5e8',
    'node': 'ac1f6b457d18',
    'url': 'http://172.30.177.1:6600'},
   {'name': 'a27193b4-a680-45cf-ad41-7b5de98b9085',
    'node': 'ac1f6b479e74',
    'url': 'http://172.30.111.183:6600'},
   {'name': '156c3875-7275-46ad-aa9b-aea1b3ee8b85',
    'node': 'ac1f6b457d18',
    'url': 'http://172.30.177.1:6600'}],
  'nsName': '03abc5b6-5f08-4470-96f8-a70487a0b5ad',
  'nsPassword': 'kz190wh0m3k1fw6dv3uz6lxe728uia5l',
  'parityShards': 2,
  'passiveS3': '',
  'reverseProxy': '',
  'storageSize': 200,
  'storageType': 'hdd',
  'tlog': {'address': '172.16.46.18:1031',
   'name': '19aaff45-9ee3-462d-9288-4f168c82536b',
   'node': 'ac1f6b457b2c',
   'url': 'http://172.30.43.86:6600'}},
 'guid': 'ca44551b-bcbc-4976-a352-afd2aa5118e1',
 'name': 'ca44551b-bcbc-4976-a352-afd2aa5118e1',
 'public': False,
 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'},
  {'category': 'status', 'state': 'ok', 'tag': 'running'},
  {'category': 'vm', 'state': 'ok', 'tag': 'running'}],
 'template': 'github.com/threefoldtech/0-templates/s3/0.0.1',
 'version': '0.0.1'}
In [21]: activevm = robot.services.get(name='dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e',template_name='dm_vm')

In [22]: activevm.data
Out[22]: 
{'data': {'configs': [],
  'cpu': 2,
  'disks': [{'diskType': 'ssd', 'label': 's3vm', 'size': 10}],
  'image': 'zero-os',
  'kernelArgs': [{'key': 'development', 'name': 'development'},
   {'key': 'zerotier', 'name': 'zerotier', 'value': 'd5e5fb6537a04d41'},
   {'key': 'ztid',
    'name': 'ztid',
    'value': 'f1c696ca0c:0:40fa1b19b0110b97d8af071d7c4d77a860fb6d03d4a6824c643b3030a86a81245b6dc743dff4264ba33c16cf95fcc3c977c153d008ee7142581edbd9af711e85:085f7b8eeea3cd18229becec7ffdefa45dfc63717825c72c72d59dff1f6ea3c9fd1b6170f381d8b9f4cf640f0b1d827696e01b5fdb73eddc47efe4f642dbdb41'}],
  'memory': 4000,
  'mgmtNic': {'id': 'd5e5fb6537a04d41',
   'type': 'zerotier',
   'ztClient': 'main'},
  'nodeId': 'ac1f6b457b2c',
  'ports': [{'name': 'minio_dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e',
    'source': 1026,
    'target': 1024}],
  'ztIdentity': 'f1c696ca0c:0:40fa1b19b0110b97d8af071d7c4d77a860fb6d03d4a6824c643b3030a86a81245b6dc743dff4264ba33c16cf95fcc3c977c153d008ee7142581edbd9af711e85:085f7b8eeea3cd18229becec7ffdefa45dfc63717825c72c72d59dff1f6ea3c9fd1b6170f381d8b9f4cf640f0b1d827696e01b5fdb73eddc47efe4f642dbdb41'},
 'guid': '29ac7a8f-32a6-4ae8-a966-84651c5a20a0',
 'name': 'dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e',
 'public': False,
 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'},
  {'category': 'status', 'state': 'ok', 'tag': 'running'}],
 'template': 'github.com/threefoldtech/0-templates/dm_vm/0.0.1',
 'version': '0.0.1'}

In [23]: tcl = j.clients.zos.get_by_id('ac1f6b457b2c')

In [24]: zrobot = tcl.containers.get('zrobot')

In [25]: zrobot.client.bash('zrobot godtoken get ').get()
Out[25]: 
STATE: 0 SUCCESS
STDOUT:
god token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdXRoZW50aWNhdGlvbiI6ImdvZF90b2tlbiJ9.Th9dT4tCekYtkjrt2kycAglOnqk6tHG5p6h6IeKIguM 

STDERR:
[Tue27 16:10] - DbConfigManager.py:104 :anager.dbconfigmanager - INFO     - using namespace ac1f6b457b2c

DATA:
In [30]: remote = j.clients.zrobot.new('test',data={'url':'http://172.30.43.86:6600','god_token_':'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdXRoZW50aWNhdGlvbiI6ImdvZF90b2tlbiJ9.Th9dT4tCekYtkjrt2kycAglOnqk
    ...: 6tHG5p6h6IeKIguM'})

In [31]: trobot = j.clients.zrobot.robots.get('test1')

In [32]: trobot = j.clients.zrobot.robots.get('test')

In [33]: disk = trobot.services.get(name='29ac7a8f-32a6-4ae8-a966-84651c5a20a0_s3vm', template_name='vdisk')

In [34]: disk.data
Out[34]: 
{'data': {'diskType': 'ssd',
  'filesystem': '',
  'label': 's3vm',
  'mountPoint': '',
  'nsName': '309b00f5-d8eb-410d-9169-84ef885ca9ca',
  'password': 'm185i4pqq3drhy0b9lsl2dxenmt3q5du',
  'size': 10,
  'zerodb': '3c1fb5c4-368a-4a13-9de8-6555019e6b9f'},
 'guid': '30d63e53-32c0-47a6-9858-50bce92b13fc',
 'name': '29ac7a8f-32a6-4ae8-a966-84651c5a20a0_s3vm',
 'public': False,
 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'},
  {'category': 'status', 'state': 'ok', 'tag': 'running'}],
 'template': 'github.com/threefoldtech/0-templates/vdisk/0.0.1',
 'version': '0.0.1'}

In [35]: zerodb = trobot.services.get(name='3c1fb5c4-368a-4a13-9de8-6555019e6b9f')

In [36]: zerodb.data
Out[36]: 
{'data': {'admin': 'gks0npzn795jelzbspgjqewnn',
  'diskType': 'ssd',
  'mode': 'user',
  'namespaces': [{'name': '309b00f5-d8eb-410d-9169-84ef885ca9ca',
    'password': 'm185i4pqq3drhy0b9lsl2dxenmt3q5du',
    'public': False,
    'size': 10}],
  'nics': [],
  'nodePort': 1025,
  'path': '/mnt/storagepools/1d50e531-cad0-4f90-94ff-4b84ec905c8d/filesystems/zdb_3c1fb5c4-368a-4a13-9de8-6555019e6b9f',
  'size': 10,
  'sync': False,
  'ztIdentity': 'f3bef7f998:0:214eff98fe597240512750f86028f039196f47f2f852192546ddfb7046c304150437c589cb2e73c52e9b503606467f08d34f53e8c65dd29a0dd271f57de3b590:256721bf6badaacc5d7d0d0cc7506aa85c3dbaa1d95f3d06ea71ff1310447ab3339f40bbd6422a1f6c20ee567824fa22e6cc2c00809c5f8530c8a1c4dc7fa679'},
 'guid': '81e4ff26-4fe1-4cc0-bc84-541390095424',
 'name': '3c1fb5c4-368a-4a13-9de8-6555019e6b9f',
 'public': False,
 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'},
  {'category': 'actions', 'state': 'ok', 'tag': 'start'},
  {'category': 'status', 'state': 'ok', 'tag': 'running'}],
 'template': 'github.com/threefoldtech/0-templates/zerodb/0.0.1',
 'version': '0.0.1'}

In [37]: tcl
Out[37]: Node <172.30.43.86:6379>

In [38]: !redis-cli -h 172.30.43.86 -p 1025 nslist
1) "default"
2) "309b00f5-d8eb-410d-9169-84ef885ca9ca"
In [40]: tcl.client.bash('lsblk | grep /mnt/storagepools/1d50e531-cad0-4f90-94ff-4b84ec905c8d').get()
Out[40]: 
STATE: 0 SUCCESS
STDOUT:
`-sdf1   8:81   0  1.8T  0 part /mnt/storagepools/1d50e531-cad0-4f90-94ff-4b84ec905c8d
In [41]: namespace = trobot.services.get(name='ee072dbe-351d-4551-9095-7ab894f52ac7')

In [42]: namespace.data
Out[42]: 
{'data': {'diskType': 'ssd',
  'mode': 'direct',
  'nsName': '03abc5b6-5f08-4470-96f8-a70487a0b5ad_tlog',
  'password': 'kz190wh0m3k1fw6dv3uz6lxe728uia5l',
  'public': False,
  'size': 10,
  'zerodb': '512048d3-762f-4143-8931-831456e60716'},
 'guid': 'ee072dbe-351d-4551-9095-7ab894f52ac7',
 'name': 'ee072dbe-351d-4551-9095-7ab894f52ac7',
 'public': False,
 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'},
  {'category': 'status', 'state': 'ok', 'tag': 'running'}],
 'template': 'github.com/threefoldtech/0-templates/namespace/0.0.1',
 'version': '0.0.1'}

In [43]: tlogzerodb = trobot.services.get(name='512048d3-762f-4143-8931-831456e60716')

In [44]: zerodb
Out[44]: robot://test/github.com/threefoldtech/0-templates/zerodb/0.0.1?name=3c1fb5c4-368a-4a13-9de8-6555019e6b9f&guid=81e4ff26-4fe1-4cc0-bc84-541390095424

In [45]: tlogzerodb.data
Out[45]: 
{'data': {'admin': 'mcedcgqp1byxpi4fke9nt22jf',
  'diskType': 'ssd',
  'mode': 'direct',
  'namespaces': [{'name': '03abc5b6-5f08-4470-96f8-a70487a0b5ad_tlog',
    'password': 'kz190wh0m3k1fw6dv3uz6lxe728uia5l',
    'public': False,
    'size': 10}],
  'nics': [],
  'nodePort': 1030,
  'path': '/mnt/storagepools/243465e4-e71f-4204-bcf9-5d212f2cae4b/filesystems/zdb_512048d3-762f-4143-8931-831456e60716',
  'size': 10,
  'sync': False,
  'ztIdentity': '7b9e2510e3:0:ce767917e3bb1af97aabc9c9e2be90baa75bbb1ec6aab2aa10722cb9924cba6b8c911c27419fd272c114387b0899a64d21ed8e479ab75f808785587d8851e96c:9a23f6ef39387e1d6c37a8bd26c6d69d4767fa9e0dad6bf3637cc87724c239a6953cee34b23578965e816876c2ddce53f5fac4a7ce0e38ef9c6e284d79f85fa2'},
 'guid': 'e52e1bd1-d421-4ec6-89e9-68bf16aef29d',
 'name': '512048d3-762f-4143-8931-831456e60716',
 'public': False,
 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'},
  {'category': 'actions', 'state': 'ok', 'tag': 'start'},
  {'category': 'status', 'state': 'ok', 'tag': 'running'}],
 'template': 'github.com/threefoldtech/0-templates/zerodb/0.0.1',
 'version': '0.0.1'}

In [46]: tcl.client.bash('lsblk | grep /mnt/storagepools/243465e4-e71f-4204-bcf9-5d212f2cae4b').get()
Out[46]: 
STATE: 0 SUCCESS
STDOUT:
`-sdi1   8:129  0  1.8T  0 part /mnt/storagepools/243465e4-e71f-4204-bcf9-5d212f2cae4b
In [47]: tcl.client.system('echo 1 > /sys/block/sdf/device/delete').get()
Out[47]: 
STATE: 0 SUCCESS
STDOUT:
1 > /sys/block/sdf/device/delete

STDERR:

DATA:

In [48]: tcl.client.bash('echo 1 > /sys/block/sdf/device/delete').get()
Out[48]: 
STATE: 0 SUCCESS
STDOUT:

STDERR:

In [51]: tcl.client.bash('lsblk | grep /mnt/storagepools/1d50e531-cad0-4f90-94ff-4b84ec905c8d').get()
Out[51]: 
STATE: 1001 ERROR
STDOUT:

STDERR:

DATA:
In [55]: tcl.client.bash('lsblk ').get()
Out[55]: 
STATE: 0 SUCCESS
STDOUT:
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  1.8T  0 disk 
`-sda1   8:1    0  1.8T  0 part /mnt/storagepools/sp_zos-cache
sdb      8:16   0  1.8T  0 disk 
`-sdb1   8:17   0  1.8T  0 part /mnt/storagepools/363b95e0-21cc-415b-af8d-c030f0ef0841
sdc      8:32   0  1.8T  0 disk 
`-sdc1   8:33   0  1.8T  0 part /mnt/storagepools/12bc2dba-2e44-4955-a2ff-cd65664e8934
sdd      8:48   0  1.8T  0 disk 
`-sdd1   8:49   0  1.8T  0 part /mnt/storagepools/6bfe7527-0a4b-44d7-ad3d-1076d36c5317
sde      8:64   0  1.8T  0 disk 
`-sde1   8:65   0  1.8T  0 part /mnt/storagepools/91d916b7-7d25-47bb-a628-7a2c1194df4c
sdg      8:96   0  1.8T  0 disk 
`-sdg1   8:97   0  1.8T  0 part /mnt/storagepools/4b44bb5f-3e2f-49e2-98d9-5ea79b168e85
sdh      8:112  0  1.8T  0 disk 
`-sdh1   8:113  0  1.8T  0 part /mnt/storagepools/62d90146-c588-4382-a2ea-c6f305d9ba2f
sdi      8:128  0  1.8T  0 disk 
`-sdi1   8:129  0  1.8T  0 part /mnt/storagepools/243465e4-e71f-4204-bcf9-5d212f2cae4b
sdj      8:144  0  1.8T  0 disk 
`-sdj1   8:145  0  1.8T  0 part /mnt/storagepools/75340fa2-3caf-4a0f-9a57-9a1fae503535
sdk      8:160  0  1.8T  0 disk 
`-sdk1   8:161  0  1.8T  0 part /mnt/storagepools/da9d3b47-ea7f-4d63-8249-9364ff509300
sdl      8:176  0  1.8T  0 disk 
`-sdl1   8:177  0  1.8T  0 part /mnt/storagepools/1678b225-326b-4a6f-9bdd-a8cf0880065d

STDERR:

DATA:

• after failure it fixed then the data shown as it

  
  In [57]: active
  Out[57]: robot://mylocal/github.com/threefoldtech/0-templates/s3/0.0.1?name=dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e&guid=dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e

In [58]: s3_redundant = robot.services.get(name='tlogs_s3_redB')

In [59]: s3_redundant.data Out[59]: {'data': {'activeS3': 'dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e', 'dataShards': 4, 'farmerIyoOrg': 'bancadati.threefold.grid.farm01', 'mgmtNic': {'id': 'd5e5fb6537a04d41', 'ztClient': 'main'}, 'minioBlockSize': 1048576, 'minioLogin': 'login', 'minioPassword': 'password', 'nsPassword': 'kz190wh0m3k1fw6dv3uz6lxe728uia5l', 'parityShards': 2, 'passiveS3': 'ca44551b-bcbc-4976-a352-afd2aa5118e1', 'reverseProxy': '', 'storageSize': 200, 'storageType': 'hdd'}, 'guid': '03abc5b6-5f08-4470-96f8-a70487a0b5ad', 'name': 'tlogs_s3_redB', 'public': False, 'state': [{'category': 'actions', 'state': 'ok', 'tag': 'install'}], 'template': 'github.com/threefoldtech/0-templates/s3_redundant/0.0.1', 'version': '0.0.1'}

- active logs 

In [92]: print (active.logs) 2018-11-27 15:53:52,293 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:274 - INFO - wait for all namespaces and vm to be installed 2018-11-27 15:53:52,293 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:434 - INFO - create namespaces to be used as a backend for minio 2018-11-27 15:53:52,294 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:436 - INFO - compute how much zerodb are required 2018-11-27 15:53:52,294 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:449 - INFO - namespaces required: 8 of 50GB 2018-11-27 15:53:52,294 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:450 - INFO - namespaces already deployed 0 2018-11-27 15:53:52,301 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:496 - INFO - create namespaces to be used as a tlog for minio 2018-11-27 15:53:52,305 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:530 - INFO - create the zero-os vm on which we will create the minio container 2018-11-27 15:54:00,417 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:596 - INFO - number of possible nodes to use for namespace deployments 119 2018-11-27 15:54:00,418 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad on node ac1f6b479ca8 2018-11-27 15:54:00,418 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad on node ac1f6b457d68 2018-11-27 15:54:00,418 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad on node ac1f6b479e74 2018-11-27 15:54:00,419 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad on node ac1f6b457bd8 2018-11-27 15:54:00,419 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad on node ac1f6b47a124 2018-11-27 15:54:00,419 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad on node ac1f6b457c5c 2018-11-27 15:54:00,420 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad on node ac1f6b457d18 2018-11-27 15:54:00,420 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad on node ac1f6b47a14c 2018-11-27 15:54:01,201 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:596 - INFO - number of possible nodes to use for namespace deployments 119 2018-11-27 15:54:01,202 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad_tlog on node ac1f6b457b2c 2018-11-27 15:54:08,781 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:524 - INFO - tlog namespaces deployed 2018-11-27 15:54:08,791 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:256 - INFO - tlog backend namespaces deployed 2018-11-27 15:54:31,632 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:464 - INFO - 1 namespaces deployed, remaining 7 2018-11-27 15:54:31,708 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:464 - INFO - 2 namespaces deployed, remaining 6 2018-11-27 15:54:33,332 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:464 - INFO - 3 namespaces deployed, remaining 5 2018-11-27 15:54:33,706 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:464 - INFO - 4 namespaces deployed, remaining 4 2018-11-27 15:54:33,718 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:464 - INFO - 5 namespaces deployed, remaining 3 2018-11-27 15:54:36,682 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:464 - INFO - 6 namespaces deployed, remaining 2 2018-11-27 15:54:39,255 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:464 - INFO - 7 namespaces deployed, remaining 1 2018-11-27 15:54:41,105 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:464 - INFO - 8 namespaces deployed, remaining 0 2018-11-27 15:54:41,121 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:250 - INFO - data backend namespaces deployed 2018-11-27 15:54:41,872 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:261 - INFO - minio vm deployed 2018-11-27 15:56:11,718 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:686 - INFO - create the minio service on the vm 2018-11-27 15:56:11,718 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:703 - INFO - wait up to 20 mins for zerorobot until it downloads the repos and starts accepting requests 2018-11-27 15:56:14,794 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:716 - INFO - vm not up yet...waiting some more 2018-11-27 15:56:26,980 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:722 - INFO - install minio 2018-11-27 15:56:43,568 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:725 - INFO - minio installed 2018-11-27 15:56:45,596 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:729 - INFO - open port 1024 on minio vm 2018-11-27 15:56:48,632 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 15:56:48,633 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 15:56:48,633 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 15:56:54,303 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 15:56:54,304 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 15:56:54,675 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 15:57:18,268 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 15:57:18,269 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 15:57:48,328 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 15:57:48,329 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 15:57:54,366 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 15:58:18,391 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 15:58:18,392 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 15:58:48,443 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 15:58:48,443 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 15:58:52,002 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 15:59:01,519 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 15:59:01,520 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 15:59:03,835 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 15:59:03,836 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 15:59:03,838 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 15:59:29,599 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 15:59:32,694 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 15:59:32,695 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 15:59:48,168 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 15:59:48,169 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 15:59:54,181 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:00:18,218 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:00:18,218 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:00:48,264 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:00:48,265 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:00:54,280 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:01:18,312 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:01:18,312 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:01:48,356 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:01:48,357 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:01:54,369 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:02:18,417 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:02:18,418 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:02:48,008 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:02:48,008 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:02:54,016 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:03:18,055 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:03:18,056 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:03:48,100 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:03:48,100 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:03:56,436 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:04:07,946 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:04:07,947 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:04:10,739 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:04:10,739 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:04:10,740 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:04:15,932 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:04:18,487 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:04:18,487 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:04:48,490 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:299 - INFO - deleting namespace ac1f6b47a124 on node http://172.30.186.54:6600 2018-11-27 16:04:51,543 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:274 - INFO - wait for all namespaces and vm to be installed 2018-11-27 16:04:51,543 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:434 - INFO - create namespaces to be used as a backend for minio 2018-11-27 16:04:51,543 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:436 - INFO - compute how much zerodb are required 2018-11-27 16:04:51,543 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:496 - INFO - create namespaces to be used as a tlog for minio 2018-11-27 16:04:51,544 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:530 - INFO - create the zero-os vm on which we will create the minio container 2018-11-27 16:04:54,759 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:449 - INFO - namespaces required: 8 of 50GB 2018-11-27 16:04:54,759 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:450 - INFO - namespaces already deployed 7 2018-11-27 16:04:57,994 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:256 - INFO - tlog backend namespaces deployed 2018-11-27 16:04:59,886 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:261 - INFO - minio vm deployed 2018-11-27 16:05:00,430 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:596 - INFO - number of possible nodes to use for namespace deployments 122 2018-11-27 16:05:00,430 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:603 - INFO - try to install namespace 03abc5b6-5f08-4470-96f8-a70487a0b5ad on node ac1f6b457d18 2018-11-27 16:05:40,001 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:464 - INFO - 8 namespaces deployed, remaining -7 2018-11-27 16:05:40,020 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:250 - INFO - data backend namespaces deployed 2018-11-27 16:05:45,733 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:686 - INFO - create the minio service on the vm 2018-11-27 16:05:45,734 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:703 - INFO - wait up to 20 mins for zerorobot until it downloads the repos and starts accepting requests 2018-11-27 16:05:46,722 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:722 - INFO - install minio 2018-11-27 16:05:48,179 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:725 - INFO - minio installed 2018-11-27 16:05:49,173 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:729 - INFO - open port 1024 on minio vm 2018-11-27 16:05:52,257 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:05:52,257 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:05:52,258 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:05:57,135 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:05:57,135 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:05:57,332 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:06:22,403 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:06:22,404 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:06:52,450 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:06:52,451 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:06:57,462 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:07:22,032 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:07:22,033 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:07:52,088 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:07:52,089 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:07:57,097 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:08:22,131 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:08:22,132 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:08:52,184 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:08:52,185 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:09:11,516 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:09:19,077 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:09:19,078 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:09:21,365 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:09:21,366 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:09:21,367 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:09:25,720 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:09:27,713 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:09:27,714 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:09:52,302 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:09:52,303 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:09:57,314 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:10:22,368 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:10:22,370 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:10:52,413 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:10:52,414 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:10:57,426 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:11:22,462 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:11:22,462 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:11:52,005 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:11:52,006 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:11:57,016 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:12:22,164 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:12:22,165 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:12:52,211 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:12:52,212 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:12:57,224 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:13:22,266 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:13:22,268 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:13:52,315 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:13:52,316 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:13:54,356 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:14:01,857 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:14:01,858 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:14:03,871 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:14:03,873 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:14:03,874 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:14:09,661 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:14:22,367 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:14:22,368 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:14:52,420 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:14:52,421 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:14:57,429 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:15:22,475 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:15:22,475 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:15:52,023 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:15:52,024 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:15:57,036 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:16:22,072 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:16:22,072 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:16:52,122 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:16:52,123 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:16:57,132 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:17:22,181 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:17:22,181 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:17:52,230 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:17:52,232 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:17:57,241 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:18:22,277 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:18:22,278 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:18:52,325 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:18:52,326 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:18:54,625 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:19:02,352 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:19:02,353 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:19:04,612 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:19:04,613 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:19:04,614 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:19:09,302 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:19:22,381 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:19:22,383 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:19:52,434 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:19:52,435 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:19:57,474 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:20:22,485 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:20:22,486 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:20:52,035 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:20:52,035 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:20:57,079 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:21:22,085 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:21:22,086 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:21:52,132 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:21:52,133 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:21:57,177 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:22:22,208 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:22:22,209 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:22:52,249 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:22:52,250 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:22:57,295 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:23:22,295 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:23:22,296 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:23:52,345 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:23:52,345 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:23:55,440 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:24:03,879 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:24:03,880 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:24:06,141 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:24:06,142 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:24:06,144 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:24:10,311 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:24:22,397 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:24:22,398 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:24:52,443 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:24:52,444 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:24:57,490 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:25:22,489 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:25:22,489 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:25:52,034 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:25:52,034 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:25:57,086 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:26:22,087 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:26:22,088 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:26:52,139 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:26:52,139 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:26:57,187 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:27:22,200 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:27:22,201 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:27:52,248 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:27:52,248 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:27:57,286 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:28:22,306 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:28:22,307 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:28:52,355 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:28:52,356 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:28:54,419 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:29:03,492 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:29:03,493 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:29:05,479 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:29:05,479 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:29:05,480 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:29:09,749 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:29:22,403 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:29:22,404 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:29:52,448 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:29:52,449 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:29:57,481 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:30:22,499 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:30:22,499 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:30:52,047 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:30:52,048 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:30:57,087 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:31:22,094 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:31:22,095 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:31:52,140 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:31:52,141 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:31:57,177 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:32:22,191 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:32:22,192 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:32:52,237 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:32:52,237 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:32:57,272 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:33:22,295 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:33:22,296 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:33:52,005 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:33:56,244 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:33:56,244 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:33:58,242 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:34:04,739 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:34:04,740 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:34:06,719 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:34:06,720 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:34:06,721 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:34:22,395 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:34:22,395 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:34:52,441 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:34:52,442 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:34:57,468 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:35:22,490 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:35:22,491 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:35:52,042 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:35:52,043 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:35:57,068 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:36:22,095 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:36:22,096 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:36:52,147 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:36:52,148 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:36:57,166 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:37:22,207 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:37:22,208 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:37:52,258 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:37:52,258 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:37:57,270 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:38:22,311 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:38:22,312 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:38:52,024 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:39:01,877 - /opt/code/github/threefoldtech/0-robot/zerorobot/template/base.py:215 - ERROR - error executing action _ensure_namespaces_connections: Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 171, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py", line 79, in create_connection raise err File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py", line 69, in create_connection sock.connect(sa) File "/usr/local/lib/python3.5/dist-packages/gevent/_socket3.py", line 335, in connect raise error(err, strerror(err)) OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 354, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.5/http/client.py", line 1106, in request self._send_request(method, url, body, headers) File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request self.endheaders(body) File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders self._send_output(message_body) File "/usr/lib/python3.5/http/client.py", line 934, in _send_output self.send(msg) File "/usr/lib/python3.5/http/client.py", line 877, in send self.connect() File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 196, in connect conn = self._new_conn() File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 180, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6be8318748>: Failed to establish a new connection: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py", line 445, in send timeout=timeout File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/local/lib/python3.5/dist-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.28.29.216', port=6600): Max retries exceeded with url: /services?template_uid=github.com%2Fthreefoldtech%2F0-templates%2Fminio%2F0.0.1&name=dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6be8318748>: Failed to establish a new connection: [Errno 113] No route to host',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute self._result = self._func() File "/opt/code/github/threefoldtech/0-templates/templates/s3/s3.py", line 128, in _ensure_namespaces_connections minio = self._minio() File "/opt/code/github/threefoldtech/0-templates/templates/s3/s3.py", line 234, in _minio return vm_robot.services.get(template_uid=MINIO_TEMPLATEUID, name=self.guid) File "/opt/code/github/threefoldtech/0-robot/zerorobot/dsl/ZeroRobotManager.py", line 117, in get results = self.find(**kwargs) File "/opt/code/github/threefoldtech/0-robot/zerorobot/dsl/ZeroRobotManager.py", line 97, in find services, = self._client.api.services.listServices(query_params=kwargs) File "/opt/code/github/threefoldtech/0-robot/JumpscaleZrobot/clients/zerorobot/client/services_service.py", line 186, in listServices resp = self.client.get(uri, None, headers, query_params, content_type) File "/opt/code/github/threefoldtech/0-robot/JumpscaleZrobot/clients/zerorobot/client/http_client.py", line 71, in get return self._handle_data(uri, data, headers, params, content_type, self.session.get) File "/opt/code/github/threefoldtech/0-robot/JumpscaleZrobot/clients/zerorobot/client/http_client.py", line 53, in _handle_data res = method(uri, headers=headers, params=params) File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 525, in get return self.request('GET', url, kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 512, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 622, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py", line 513, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.28.29.216', port=6600): Max retries exceeded with url: /services?template_uid=github.com%2Fthreefoldtech%2F0-templates%2Fminio%2F0.0.1&name=dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6be8318748>: Failed to establish a new connection: [Errno 113] No route to host',))

2018-11-27 16:39:01,892 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:39:01,892 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:39:06,489 - /opt/code/github/threefoldtech/0-robot/zerorobot/template/base.py:215 - ERROR - error executing action _monitor_minio: Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 171, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py", line 79, in create_connection raise err File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py", line 69, in create_connection sock.connect(sa) File "/usr/local/lib/python3.5/dist-packages/gevent/_socket3.py", line 335, in connect raise error(err, strerror(err)) OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 354, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.5/http/client.py", line 1106, in request self._send_request(method, url, body, headers) File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request self.endheaders(body) File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders self._send_output(message_body) File "/usr/lib/python3.5/http/client.py", line 934, in _send_output self.send(msg) File "/usr/lib/python3.5/http/client.py", line 877, in send self.connect() File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 196, in connect conn = self._new_conn() File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 180, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6be83f98d0>: Failed to establish a new connection: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py", line 445, in send timeout=timeout File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/local/lib/python3.5/dist-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.28.29.216', port=6600): Max retries exceeded with url: /services?template_uid=github.com%2Fthreefoldtech%2F0-templates%2Fminio%2F0.0.1&name=dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6be83f98d0>: Failed to establish a new connection: [Errno 113] No route to host',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/code/github/threefoldtech/0-robot/zerorobot/task/task.py", line 78, in execute self._result = self._func() File "/opt/code/github/threefoldtech/0-templates/templates/s3/s3.py", line 654, in _monitor_minio minio = vm_robot.services.get(template_uid=MINIO_TEMPLATEUID, name=self.guid) File "/opt/code/github/threefoldtech/0-robot/zerorobot/dsl/ZeroRobotManager.py", line 117, in get results = self.find(**kwargs) File "/opt/code/github/threefoldtech/0-robot/zerorobot/dsl/ZeroRobotManager.py", line 97, in find services, = self._client.api.services.listServices(query_params=kwargs) File "/opt/code/github/threefoldtech/0-robot/JumpscaleZrobot/clients/zerorobot/client/services_service.py", line 186, in listServices resp = self.client.get(uri, None, headers, query_params, content_type) File "/opt/code/github/threefoldtech/0-robot/JumpscaleZrobot/clients/zerorobot/client/http_client.py", line 71, in get return self._handle_data(uri, data, headers, params, content_type, self.session.get) File "/opt/code/github/threefoldtech/0-robot/JumpscaleZrobot/clients/zerorobot/client/http_client.py", line 53, in _handle_data res = method(uri, headers=headers, params=params) File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 525, in get return self.request('GET', url, kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 512, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 622, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py", line 513, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.28.29.216', port=6600): Max retries exceeded with url: /services?template_uid=github.com%2Fthreefoldtech%2F0-templates%2Fminio%2F0.0.1&name=dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6be83f98d0>: Failed to establish a new connection: [Errno 113] No route to host',))

2018-11-27 16:39:06,490 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:39:22,453 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:39:22,453 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:39:52,497 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:39:52,497 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:39:57,007 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:40:22,045 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:40:22,046 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:40:52,098 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:40:52,099 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:40:57,108 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:41:22,145 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:41:22,146 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:41:52,196 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:41:52,196 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:41:57,208 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:42:22,273 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:42:22,274 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:42:52,325 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:42:52,326 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:42:57,337 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:43:22,379 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:43:22,380 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:43:54,063 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:44:00,358 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:44:00,359 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:44:02,356 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:44:02,357 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:44:02,358 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:44:08,420 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:44:08,420 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:44:08,420 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:44:22,489 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:44:22,490 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:44:52,038 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:44:52,039 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:44:57,050 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:45:22,093 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:45:22,093 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:45:52,148 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:45:52,149 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:45:57,159 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:46:22,191 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:46:22,191 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:46:52,244 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:46:52,245 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:46:57,257 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:47:22,310 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:47:22,310 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:47:52,373 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:47:52,373 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:47:57,384 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:48:22,423 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:48:22,425 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:48:57,861 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:49:05,406 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:49:05,407 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:49:07,608 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:49:07,609 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:49:07,611 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:49:14,139 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:49:14,140 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:49:14,141 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:49:22,075 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:49:22,076 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:49:52,134 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:49:52,135 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:49:57,143 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:50:22,182 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:50:22,183 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:50:52,232 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:50:52,233 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:50:57,241 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:51:22,289 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:51:22,290 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:51:52,341 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:51:52,342 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:51:57,352 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:52:22,423 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:52:22,424 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:52:52,472 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:52:52,473 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:52:57,483 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:53:22,022 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:53:22,023 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:53:52,073 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:53:52,074 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:53:55,177 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:54:01,608 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:54:01,609 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:54:03,848 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:54:03,848 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:54:03,848 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:54:08,141 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:54:22,188 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:54:22,189 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:54:52,236 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:54:52,237 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:54:57,249 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:55:22,283 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:55:22,283 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:55:52,332 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:55:52,332 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:55:57,344 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:56:22,387 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:56:22,388 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:56:52,433 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:56:52,433 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:56:57,443 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:57:22,482 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:57:22,482 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:57:52,028 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:57:52,029 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:57:57,041 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:58:22,083 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:58:22,084 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:58:52,135 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:58:52,137 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:58:54,218 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 16:59:00,803 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 16:59:00,804 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 16:59:02,900 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 16:59:02,901 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 16:59:02,902 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 16:59:07,140 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 16:59:22,190 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:59:22,191 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:59:52,242 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 16:59:52,244 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 16:59:57,255 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 17:00:22,291 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:00:22,292 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:00:52,344 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:00:52,345 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:00:57,355 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 17:01:22,395 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:01:22,397 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:01:52,447 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:01:52,448 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:01:57,459 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 17:02:22,495 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:02:22,496 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:02:52,048 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:02:52,049 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:02:57,066 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 17:03:22,095 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:03:22,096 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:03:52,145 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:03:52,146 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:03:54,231 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:116 - INFO - verify data backend namespace connections 2018-11-27 17:04:01,804 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:131 - INFO - namespace connection in service data are in sync with reality 2018-11-27 17:04:01,805 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:138 - INFO - verify tlog namespace connections 2018-11-27 17:04:04,483 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:152 - INFO - tlog namespace connection in service data is in sync with reality 2018-11-27 17:04:04,484 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:154 - INFO - verify master namespace connections 2018-11-27 17:04:04,486 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:73 - INFO - update minio urls 2018-11-27 17:04:08,982 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 17:04:22,196 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:04:22,197 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:04:52,247 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:04:52,247 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:04:57,267 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 17:05:22,302 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:05:22,303 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:05:52,354 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:05:52,355 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:05:57,367 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 17:06:22,403 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:06:22,403 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:06:52,454 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:06:52,455 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:06:57,468 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e 2018-11-27 17:07:22,003 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:07:22,004 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:07:52,054 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:176 - INFO - Monitor minio vm disk 2018-11-27 17:07:52,055 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:187 - INFO - Monitor minio vm 2018-11-27 17:07:57,065 - /opt/code/github/threefoldtech/0-templates/templates/s3/s3.py:201 - INFO - Monitor s3 dc1c3c1f-1d0c-4f7c-bbf9-636f3a1bcb7e

[ramezsaeed]

@john-kheir the loop happened again .. but this time the vdisk used different ssd from the tlog

https://gist.github.com/john-kheir/52bb31a2ecf01e1bf79933f707ff19b8

[ramezsaeed]

duplicate https://github.com/threefoldtech/0-templates/issues/220