Verify whether current recover and failures work

nishantrai18 commented 4 years ago

Took a look at the logs of kills and recovery. This doesn't seem very random.

Processes: dict_values([ProcessSpec(name='process-0', port=40000), ProcessSpec(name='process-1', port=40001), ProcessSpec(name='process-2', port=40002), ProcessSpec(name='process-3', port=40003), ProcessSpec(name='process-4', port=40004), ProcessSpec(name='process-5', port=40005), ProcessSpec(name='process-6', port=40006), ProcessSpec(name='process-7', port=40007), ProcessSpec(name='process-8', port=40008)])
CRITICAL  17:36:14 MainThread   main.py:147::main                 | All nodes started
CRITICAL  17:36:22 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-GGIRT342), qty:1) which is a finale item
CRITICAL  17:36:22 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-LSHYB274), qty:1) which is a finale item
CRITICAL  17:36:22 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-ISJRL732), qty:1) which is a finale item
CRITICAL  17:36:25 ops-runner   ops_generator.py:49::kill_node            | Node 0 to be killed
CRITICAL  17:36:26 ops-runner   ops_generator.py:59::recover_node         | Node 0 to be recovered
CRITICAL  17:36:27 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-VBDWL309), qty:1) which is a finale item
CRITICAL  17:36:27 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-NDPVJ921), qty:1) which is a finale item
CRITICAL  17:36:27 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-GQMZG557), qty:1) which is a finale item
CRITICAL  17:36:29 ops-runner   ops_generator.py:49::kill_node            | Node 0 to be killed
CRITICAL  17:36:32 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-PQVNA139), qty:1) which is a finale item
CRITICAL  17:36:32 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-ERBOS404), qty:1) which is a finale item
CRITICAL  17:36:32 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-UCLUF396), qty:1) which is a finale item
CRITICAL  17:36:35 ops-runner   ops_generator.py:59::recover_node         | Node 0 to be recovered
CRITICAL  17:36:37 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-BJHGU174), qty:1) which is a finale item
CRITICAL  17:36:37 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-IDOBC795), qty:1) which is a finale item
CRITICAL  17:36:37 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-QVGZA214), qty:1) which is a finale item
CRITICAL  17:36:42 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-NMVDE808), qty:1) which is a finale item
CRITICAL  17:36:42 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-FESMC862), qty:1) which is a finale item
CRITICAL  17:36:42 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-VKKYN992), qty:1) which is a finale item
CRITICAL  17:36:44 ops-runner   ops_generator.py:49::kill_node            | Node 4 to be killed
CRITICAL  17:36:47 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-WJRHF444), qty:1) which is a finale item
CRITICAL  17:36:47 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-JKMAX995), qty:1) which is a finale item
CRITICAL  17:36:47 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-AZWFT847), qty:1) which is a finale item
CRITICAL  17:36:50 ops-runner   ops_generator.py:49::kill_node            | Node 4 to be killed
CRITICAL  17:36:52 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-DHBZU333), qty:1) which is a finale item
CRITICAL  17:36:52 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-KHWKV575), qty:1) which is a finale item
CRITICAL  17:36:52 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-OIACZ997), qty:1) which is a finale item
CRITICAL  17:36:56 ops-runner   ops_generator.py:49::kill_node            | Node 7 to be killed
CRITICAL  17:36:57 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-LICVS969), qty:1) which is a finale item
CRITICAL  17:36:57 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-WQSOS930), qty:1) which is a finale item
CRITICAL  17:36:57 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-YYDKQ184), qty:1) which is a finale item
CRITICAL  17:37:02 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-YRZGG591), qty:1) which is a finale item
CRITICAL  17:37:02 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-XAQOS741), qty:1) which is a finale item
CRITICAL  17:37:02 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-ZCKKS684), qty:1) which is a finale item
CRITICAL  17:37:04 ops-runner   ops_generator.py:59::recover_node         | Node 4 to be recovered
CRITICAL  17:37:06 ops-runner   ops_generator.py:59::recover_node         | Node 4 to be recovered
CRITICAL  17:37:07 ops-runner   ops_generator.py:59::recover_node         | Node 7 to be recovered
CRITICAL  17:37:07 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-KVNQY204), qty:1) which is a finale item
CRITICAL  17:37:07 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-SRFEW368), qty:1) which is a finale item
CRITICAL  17:37:07 production-stage-4 sc_stage.py:407::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:2, id:2-DOQRV852), qty:1) which is a finale item
CRITICAL  17:37:12 production-stage-7 sc_stage.py:407::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:4, id:4-RIWHG675), qty:1) which is a finale item
CRITICAL  17:37:12 production-stage-2 sc_stage.py:407::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-PGEVW928), qty:1) which is a finale item

chuanqichen commented 4 years ago

pythonw main.py --num_types 7 --nodes_per_type 5 --run_test_ops True --failure_rate 10 --recover_rate 10 --update_dep_rate 3 --log_level Critical CRITICAL 21:07:32 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:07:33 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:07:36 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:07:38 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:07:44 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:07:45 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:07:48 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:07:50 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:07:56 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:07:57 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:08:02 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:08:03 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:08:05 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:08:14 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:08:15 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:08:17 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:08:20 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:08:20 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:08:26 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:08:27 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:08:28 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:08:28 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:08:31 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:08:37 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:08:38 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:08:42 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:08:44 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:08:46 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:08:46 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:08:48 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:08:54 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:08:56 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:08:57 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:09:00 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:09:03 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:09:04 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:09:11 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:09:13 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:09:15 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:09:17 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:09:20 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:09:21 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:09:23 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:09:26 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:09:27 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:09:34 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:09:35 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:09:36 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:09:40 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:09:42 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:09:45 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:09:45 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:09:47 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:09:48 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:09:51 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:09:52 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:10:01 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:10:02 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:10:03 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:10:09 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:10:12 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:10:19 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:10:22 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:10:28 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:10:30 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:10:32 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:10:33 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:10:36 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:10:39 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:10:40 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:10:41 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:10:42 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:10:45 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:10:46 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:10:49 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:10:50 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:10:51 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:10:54 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:10:55 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:11:02 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:11:18 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:11:18 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:11:19 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:11:19 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:11:21 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:11:23 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:11:25 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:11:26 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:11:28 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:11:30 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:11:32 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:11:37 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:11:39 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:11:40 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:11:42 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:11:44 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:11:47 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:11:47 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:11:48 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:11:56 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:11:58 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:11:59 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:12:03 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:12:05 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:12:08 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:12:08 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:12:12 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:12:14 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:12:21 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:12:28 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:12:36 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:12:46 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:12:50 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:13:05 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:13:06 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:13:13 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:13:13 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:13:14 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:13:23 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:13:27 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:13:28 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:13:33 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:13:34 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:13:42 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:13:45 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:13:50 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:13:56 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:13:58 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:13:59 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:14:08 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:14:10 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:14:13 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:14:15 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:14:16 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:14:20 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:14:21 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:14:25 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:14:31 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:14:32 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:14:34 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:14:36 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:14:49 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:14:51 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:14:52 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:14:53 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:14:55 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:15:01 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:15:11 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:15:17 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:15:20 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:15:21 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:15:27 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:15:28 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:15:32 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:15:36 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:15:38 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:15:44 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:15:47 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:15:47 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:15:48 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:15:50 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:15:51 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:15:55 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:16:02 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:16:05 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:16:07 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:16:08 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:16:09 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:16:13 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:16:17 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:16:18 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:16:20 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:16:21 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:16:23 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:16:24 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:16:31 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:16:32 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:16:32 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:16:42 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:16:48 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:16:51 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:16:51 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:16:54 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:16:56 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:16:57 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:16:57 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:17:00 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:17:02 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:17:05 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:17:10 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:17:13 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:17:16 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:17:21 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:17:24 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:17:29 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:17:35 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:17:46 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:17:49 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:17:52 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:17:54 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:17:56 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:17:58 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:18:07 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:18:10 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:18:10 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:18:11 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:18:13 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:18:15 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:18:22 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:18:26 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:18:28 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:18:30 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:18:31 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:18:34 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:18:36 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:18:37 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:18:38 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:18:46 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:18:49 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:18:53 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:19:03 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:19:04 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:19:06 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:19:10 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:19:15 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:19:19 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:19:21 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:19:28 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:19:30 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:19:34 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:19:34 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:19:35 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:19:37 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:19:38 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:19:46 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:19:57 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:20:00 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:20:04 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:20:10 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:20:14 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:20:15 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:20:17 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:20:18 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:20:24 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:20:24 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:20:25 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:20:30 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:20:37 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:20:38 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:20:41 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:20:42 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:20:42 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:20:45 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:20:46 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:20:47 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:20:49 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:20:53 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:20:54 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:20:56 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:20:59 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:21:02 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:21:03 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:21:04 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:21:05 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:21:05 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:21:07 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:21:08 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:21:12 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:21:16 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:21:20 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:21:25 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:21:28 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:21:36 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:21:37 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:21:37 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:21:44 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:21:46 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:21:48 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:21:51 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:21:52 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:21:53 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:21:56 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:22:04 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:22:09 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:22:12 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:22:15 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:22:15 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:22:22 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:22:28 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:22:29 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:22:30 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:22:34 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:22:35 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:22:37 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:22:45 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:22:46 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:22:50 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:22:52 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:22:57 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:23:02 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:23:06 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:23:07 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:23:17 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:23:17 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:23:22 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:23:23 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:23:29 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:23:29 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:23:31 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:23:41 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:23:44 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:23:47 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:23:47 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:23:48 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:23:50 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:23:52 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:23:53 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:23:55 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:23:58 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:24:02 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:24:03 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:24:08 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:24:11 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:24:14 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:24:17 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:24:20 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:24:26 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:24:31 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:24:32 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:24:32 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:24:35 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:24:41 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:24:44 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:24:46 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:24:54 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:25:02 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:25:03 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:25:03 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:25:06 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:25:09 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:25:14 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:25:14 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:25:21 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:25:28 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:25:29 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:25:35 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:25:36 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:25:40 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:25:41 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:25:43 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:25:45 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:25:51 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:25:53 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:25:57 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:25:58 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:26:04 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:26:04 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:26:06 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:26:07 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:26:10 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:26:13 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:26:25 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:26:28 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:26:29 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:26:29 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:26:37 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:26:37 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:26:38 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:26:38 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:26:39 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:26:52 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:26:58 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:27:00 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:27:05 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:27:07 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:27:12 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:27:16 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:27:17 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:27:19 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:27:20 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:27:22 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:27:26 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:27:28 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:27:29 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:27:30 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:27:32 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:27:34 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:27:37 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:27:40 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:27:40 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:27:41 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:27:47 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:27:48 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:27:50 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:27:52 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:27:54 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:28:04 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:28:04 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:28:06 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:28:07 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:28:15 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:28:17 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:28:24 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:28:25 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:28:26 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:28:26 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:28:27 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:28:32 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:28:40 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:28:40 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:28:41 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:28:44 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:28:47 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:28:50 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:28:54 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:29:02 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:29:02 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:29:03 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:29:06 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:29:07 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:29:10 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:29:12 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:29:13 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:29:13 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:29:16 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:29:21 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:29:22 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:29:27 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:29:33 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:29:39 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:29:42 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:29:42 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:29:43 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:29:45 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:29:47 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:29:51 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:29:53 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:29:58 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:30:04 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:30:06 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:30:11 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:30:12 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:30:13 ops-runner ops_generator.py:59::recover_node | Node 4 to be recovered CRITICAL 21:30:15 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:30:19 ops-runner ops_generator.py:59::recover_node | Node 3 to be recovered CRITICAL 21:30:21 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:30:24 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:30:25 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:30:25 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:30:27 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:30:27 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:30:28 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:30:28 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:30:32 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:30:32 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:30:34 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:30:48 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:30:48 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:30:50 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:30:53 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:30:58 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:31:03 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:31:05 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:31:06 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:31:07 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:31:08 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:31:16 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:31:17 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:31:20 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:31:25 ops-runner ops_generator.py:59::recover_node | Node 11 to be recovered CRITICAL 21:31:26 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:31:27 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:31:30 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:31:31 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:31:35 ops-runner ops_generator.py:49::kill_node | Node 6 to be killed CRITICAL 21:31:43 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:31:48 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:31:54 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:31:55 ops-runner ops_generator.py:49::kill_node | Node 0 to be killed CRITICAL 21:31:58 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:32:01 ops-runner ops_generator.py:49::kill_node | Node 11 to be killed CRITICAL 21:32:02 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:32:02 ops-runner ops_generator.py:49::kill_node | Node 3 to be killed CRITICAL 21:32:08 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:32:15 ops-runner ops_generator.py:59::recover_node | Node 10 to be recovered CRITICAL 21:32:17 ops-runner ops_generator.py:59::recover_node | Node 5 to be recovered CRITICAL 21:32:17 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:32:18 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:32:21 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:32:24 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:32:27 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:32:34 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered CRITICAL 21:32:34 ops-runner ops_generator.py:49::kill_node | Node 8 to be killed CRITICAL 21:32:36 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:32:37 ops-runner ops_generator.py:49::kill_node | Node 7 to be killed CRITICAL 21:32:40 ops-runner ops_generator.py:59::recover_node | Node 8 to be recovered CRITICAL 21:32:40 ops-runner ops_generator.py:49::kill_node | Node 10 to be killed CRITICAL 21:32:45 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:32:45 ops-runner ops_generator.py:49::kill_node | Node 2 to be killed CRITICAL 21:32:47 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:32:48 ops-runner ops_generator.py:49::kill_node | Node 5 to be killed CRITICAL 21:32:49 ops-runner ops_generator.py:49::kill_node | Node 4 to be killed CRITICAL 21:32:50 ops-runner ops_generator.py:59::recover_node | Node 7 to be recovered CRITICAL 21:32:56 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:32:56 ops-runner ops_generator.py:49::kill_node | Node 9 to be killed CRITICAL 21:33:01 ops-runner ops_generator.py:59::recover_node | Node 9 to be recovered CRITICAL 21:33:04 ops-runner ops_generator.py:59::recover_node | Node 0 to be recovered CRITICAL 21:33:05 ops-runner ops_generator.py:59::recover_node | Node 1 to be recovered CRITICAL 21:33:12 ops-runner ops_generator.py:59::recover_node | Node 2 to be recovered CRITICAL 21:33:15 ops-runner ops_generator.py:49::kill_node | Node 1 to be killed CRITICAL 21:33:27 ops-runner ops_generator.py:59::recover_node | Node 6 to be recovered

chuanqichen commented 4 years ago

@nishantrai18 @raokrutarth If the failure rate very low, then the failure and recover will work more sequentially one node after another. The current random is over one minute. I could stretch it to say 15 minutes or longer time, then it might work better. Would you want to try that?

chuanqichen commented 4 years ago

I show logs of one of my testing above with higher failure rates, it is more random than lower failure rate.

raokrutarth commented 4 years ago

@nishantrai18 how random do we expect it?

raokrutarth / manufacturing-os

Verify whether current recover and failures work #123