search

raokrutarth / manufacturing-os

A POC distributed operating system for geo-seperated automotive manufacturing
Creative Commons Zero v1.0 Universal
1 stars 0 forks source link

Add stage WAL based recovery #113

Closed raokrutarth closed 4 years ago

raokrutarth commented 4 years ago

Add:

Inbound recovery

outbound recovery

raokrutarth commented 4 years ago

Logs showing all stages manufacture correctly and receive delivery confirmations from downstream:

zee@zee-Z620:~/sharp/manufacturing-os$ cat tmp/manufacturing-os.log | grep BatchDeliveryConfirm
INFO    00:26:41 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-NYWGE155), quantity:1), req-id:788e4ce1) from node 5
INFO    00:26:43 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-UQOLD547), quantity:1), req-id:38d0a067) from node 13
INFO    00:26:47 subscriber-6 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:6, to:5, batch:ItemReq(Item(type:4, id:4-QUOCQ953), quantity:1), req-id:f50a1ce6) from node 6
INFO    00:26:48 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-MEXUS961), quantity:1), req-id:16009973) from node 13
INFO    00:26:51 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-KFNRZ701), quantity:1), req-id:5b04888f) from node 5
INFO    00:26:54 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:13, batch:ItemReq(Item(type:7, id:7-FJBQA440), quantity:1), req-id:3389b77e) from node 3
INFO    00:26:55 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-SCBKV513), quantity:1), req-id:15a926d4) from node 5
INFO    00:26:56 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-LTUHH904), quantity:1), req-id:3d62e974) from node 13
INFO    00:26:57 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:13, batch:ItemReq(Item(type:7, id:7-SJZLE972), quantity:1), req-id:216325a2) from node 3
INFO    00:27:00 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-BTUAI481), quantity:1), req-id:f4a78e3d) from node 13
INFO    00:27:01 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:5, batch:ItemReq(Item(type:4, id:4-UGWFG260), quantity:1), req-id:ee6c15f7) from node 3
INFO    00:27:02 subscriber-10 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:10, to:5, batch:ItemReq(Item(type:4, id:4-WDWIM618), quantity:1), req-id:b30c2a38) from node 10
INFO    00:27:03 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-HDFXL481), quantity:1), req-id:77ae6e1d) from node 5
INFO    00:27:06 subscriber-1 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:1, to:13, batch:ItemReq(Item(type:7, id:7-DGZUB399), quantity:1), req-id:13ee4247) from node 1
INFO    00:27:06 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-SKXTM144), quantity:1), req-id:c17b06f2) from node 5
INFO    00:27:09 subscriber-1 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:1, to:13, batch:ItemReq(Item(type:7, id:7-ZRZTT015), quantity:1), req-id:cb54681d) from node 1
INFO    00:27:11 subscriber-10 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:10, to:5, batch:ItemReq(Item(type:4, id:4-RQZVZ263), quantity:1), req-id:a671d8f4) from node 10
INFO    00:27:13 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-JBYLR252), quantity:1), req-id:b8e81306) from node 13
INFO    00:27:13 subscriber-4 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:4, to:5, batch:ItemReq(Item(type:4, id:4-GWHDE834), quantity:1), req-id:e544d96d) from node 4
INFO    00:27:15 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-GXJZV016), quantity:1), req-id:b1474e2d) from node 5
INFO    00:27:16 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-VFTSZ791), quantity:1), req-id:2663325c) from node 13
INFO    00:27:16 subscriber-15 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:15, to:1, batch:ItemReq(Item(type:1, id:1-NSPFH002), quantity:1), req-id:5a0f1eb7) from node 15
INFO    00:27:17 subscriber-4 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:4, to:1, batch:ItemReq(Item(type:1, id:1-DEPLE044), quantity:1), req-id:1241f22c) from node 4
INFO    00:27:21 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-UUMLF056), quantity:1), req-id:820be86c) from node 5
INFO    00:27:21 subscriber-10 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:10, to:13, batch:ItemReq(Item(type:7, id:7-OWCKZ333), quantity:1), req-id:2c8cbf7c) from node 10
INFO    00:27:22 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:4, batch:ItemReq(Item(type:3, id:3-ZQTXW710), quantity:1), req-id:13fc484d) from node 3
INFO    00:27:23 subscriber-1 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:1, to:13, batch:ItemReq(Item(type:7, id:7-TRRJV401), quantity:1), req-id:37cf180c) from node 1
INFO    00:27:24 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-FPJGT430), quantity:1), req-id:7e1665a7) from node 5
INFO    00:27:24 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-TDHZC042), quantity:1), req-id:2369efc2) from node 13
INFO    00:27:27 subscriber-4 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:4, to:1, batch:ItemReq(Item(type:1, id:1-VWVOI230), quantity:1), req-id:1be42490) from node 4
INFO    00:27:28 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-IJJQX459), quantity:1), req-id:6bd30727) from node 13
INFO    00:27:29 subscriber-15 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:15, to:3, batch:ItemReq(Item(type:2, id:2-LBUZI841), quantity:1), req-id:1ad1dfe6) from node 15
INFO    00:27:32 subscriber-1 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:1, to:13, batch:ItemReq(Item(type:7, id:7-RGJLS602), quantity:1), req-id:12bb72c3) from node 1
INFO    00:27:34 subscriber-4 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:4, to:5, batch:ItemReq(Item(type:4, id:4-VYSJW049), quantity:1), req-id:1330ed3c) from node 4
INFO    00:27:34 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-HEIRR146), quantity:1), req-id:552f9f0a) from node 5
INFO    00:27:35 subscriber-1 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:1, to:13, batch:ItemReq(Item(type:7, id:7-EKSSK051), quantity:1), req-id:5e87be5c) from node 1
INFO    00:27:36 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-GRHNM015), quantity:1), req-id:4cfd1d4b) from node 13
INFO    00:27:37 subscriber-4 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:4, to:5, batch:ItemReq(Item(type:4, id:4-GGBKJ553), quantity:1), req-id:a7813d8f) from node 4
INFO    00:27:37 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-VSXLT016), quantity:1), req-id:06ebfc51) from node 5
INFO    00:27:41 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-WOTYS316), quantity:1), req-id:36a95681) from node 13
INFO    00:27:41 subscriber-4 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:4, to:5, batch:ItemReq(Item(type:4, id:4-XXHFB961), quantity:1), req-id:576aebf6) from node 4
INFO    00:27:42 subscriber-10 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:10, to:1, batch:ItemReq(Item(type:1, id:1-EJFQY728), quantity:1), req-id:0d615dea) from node 10
INFO    00:27:43 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:4, batch:ItemReq(Item(type:3, id:3-UQFNG101), quantity:1), req-id:8f486f5f) from node 3
INFO    00:27:45 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-TIHRI453), quantity:1), req-id:a7d295bb) from node 5
INFO    00:27:46 subscriber-10 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:10, to:1, batch:ItemReq(Item(type:1, id:1-QVPMI356), quantity:1), req-id:069702b6) from node 10
INFO    00:27:46 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:5, batch:ItemReq(Item(type:4, id:4-PKOVZ900), quantity:1), req-id:c2fa903f) from node 3
INFO    00:27:47 subscriber-1 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:1, to:13, batch:ItemReq(Item(type:7, id:7-ENZBO045), quantity:1), req-id:5ffbb123) from node 1
INFO    00:27:49 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:5, batch:ItemReq(Item(type:4, id:4-KIVPB864), quantity:1), req-id:9565a4cd) from node 3
INFO    00:27:49 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-THIUC579), quantity:1), req-id:ba837786) from node 5
INFO    00:27:50 subscriber-1 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:1, to:13, batch:ItemReq(Item(type:7, id:7-NWIBU110), quantity:1), req-id:98e56d1b) from node 1
INFO    00:27:51 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-LXLLL526), quantity:1), req-id:fb184f48) from node 13
INFO    00:27:53 subscriber-6 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:6, to:3, batch:ItemReq(Item(type:2, id:2-QGPRG467), quantity:1), req-id:1c4655ca) from node 6
INFO    00:27:54 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:5, batch:ItemReq(Item(type:4, id:4-YIIKE629), quantity:1), req-id:eaaaf3a3) from node 3
INFO    00:27:55 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-CLIEQ195), quantity:1), req-id:24939b04) from node 13
INFO    00:27:56 subscriber-4 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:4, to:1, batch:ItemReq(Item(type:1, id:1-QZMRH744), quantity:1), req-id:68c37e0d) from node 4
INFO    00:27:59 subscriber-4 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:4, to:1, batch:ItemReq(Item(type:1, id:1-PZITI747), quantity:1), req-id:546e0ad2) from node 4
INFO    00:28:00 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-RZFMA030), quantity:1), req-id:5fdf1b14) from node 5
INFO    00:28:01 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-VQWLA919), quantity:1), req-id:431862a9) from node 13
INFO    00:28:02 subscriber-15 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:15, to:6, batch:ItemReq(Item(type:5, id:5-DGJHM115), quantity:1), req-id:960f2740) from node 15
INFO    00:28:02 subscriber-10 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:10, to:4, batch:ItemReq(Item(type:3, id:3-DYYSI309), quantity:1), req-id:187b64db) from node 10
INFO    00:28:05 subscriber-6 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:6, to:5, batch:ItemReq(Item(type:4, id:4-QTJAH480), quantity:1), req-id:4a6ec529) from node 6
INFO    00:28:05 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-QHJOV317), quantity:1), req-id:ad6883ba) from node 5
INFO    00:28:06 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:13, batch:ItemReq(Item(type:7, id:7-GYCNE439), quantity:1), req-id:a3182a53) from node 3
INFO    00:28:06 subscriber-15 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:15, to:4, batch:ItemReq(Item(type:3, id:3-GCHXL973), quantity:1), req-id:746511a9) from node 15
INFO    00:28:06 subscriber-1 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:1, to:13, batch:ItemReq(Item(type:7, id:7-JFTUT998), quantity:1), req-id:982f41c7) from node 1
INFO    00:28:07 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-AENJU838), quantity:1), req-id:701f06a3) from node 13
INFO    00:28:08 subscriber-5 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:5, to:0, batch:ItemReq(Item(type:0, id:0-LHQYX665), quantity:1), req-id:b7f2f0f5) from node 5
INFO    00:28:09 subscriber-3 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:3, to:13, batch:ItemReq(Item(type:7, id:7-PGTTV004), quantity:1), req-id:2e1052fb) from node 3
INFO    00:28:11 subscriber-13 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:13, to:0, batch:ItemReq(Item(type:0, id:0-ORLWC011), quantity:1), req-id:101ca0b8) from node 13
INFO    00:28:12 subscriber-6 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:6, to:5, batch:ItemReq(Item(type:4, id:4-LQGDK991), quantity:1), req-id:0bd4073b) from node 6
INFO    00:28:12 subscriber-4 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:4, to:1, batch:ItemReq(Item(type:1, id:1-VZPTS509), quantity:1), req-id:c107e578) from node 4
INFO    00:28:12 subscriber-10 messages.py:341::sendMessage          | sending message BatchDeliveryConfirmResponse(from:10, to:15, batch:ItemReq(Item(type:8, id:8-ZVQNR153), quantity:1), req-id:c4819e7b) from node 10
zee@zee-Z620:~/sharp/manufacturing-os$ 
zee@zee-Z620:~/sharp/manufacturing-os$ cat tmp/manufacturing-os.log | grep BatchDeliveryConfirm | wc -l
77
raokrutarth commented 4 years ago

I'd prefer to merge @chuanqichen 's changes with him on a call. is there another way I can simulate testing this?

nishantrai18 commented 4 years ago

I meant the changes already in master. You can use high failure and recovery rates.

nishantrai18 commented 4 years ago

Use something like, python main.py --run_test_ops False --log_level warning --num_types 6 --nodes_per_type 3 --complexity high --failure_rate 10 --recover_rate 20

nishantrai18 commented 4 years ago

Also, modify manufacture time delay to be a large value to have multiple items in transit

raokrutarth commented 4 years ago

ok. running /usr/bin/python3 src/main.py --num_types 5 --nodes_per_type 3 --failure_rate 4 --recover_rate 3

raokrutarth commented 4 years ago

Ran with command: /usr/bin/python3 src/main.py --num_types 5 --nodes_per_type 3 --failure_rate 4 --recover_rate 3

Logs for recovery:

zee@zee-Z620:~/sharp/manufacturing-os$ cat tmp/manufacturing-os.log | grep -i wal
INFO    01:22:33 MainThread   sc_stage.py:87::_attempt_log_recovery | Node 1's batch status WALs are empty. Skipping post-crash item and material status verification
INFO    01:22:33 MainThread   sc_stage.py:87::_attempt_log_recovery | Node 2's batch status WALs are empty. Skipping post-crash item and material status verification
INFO    01:22:33 MainThread   sc_stage.py:87::_attempt_log_recovery | Node 0's batch status WALs are empty. Skipping post-crash item and material status verification
INFO    01:22:33 MainThread   sc_stage.py:87::_attempt_log_recovery | Node 3's batch status WALs are empty. Skipping post-crash item and material status verification
INFO    01:22:33 MainThread   sc_stage.py:87::_attempt_log_recovery | Node 4's batch status WALs are empty. Skipping post-crash item and material status verification
INFO    01:22:33 MainThread   sc_stage.py:87::_attempt_log_recovery | Node 5's batch status WALs are empty. Skipping post-crash item and material status verification
INFO    01:22:33 MainThread   sc_stage.py:87::_attempt_log_recovery | Node 6's batch status WALs are empty. Skipping post-crash item and material status verification
INFO    01:22:40 subscriber-5 sc_stage.py:293::process_batch_request_response | Node 5 marking batch ItemReq(Item(type:0, id:0-FFLGZ803), qty:1) in-transit in local WAL
INFO    01:22:45 subscriber-5 sc_stage.py:293::process_batch_request_response | Node 5 marking batch ItemReq(Item(type:0, id:0-ZEWEE371), qty:1) in-transit in local WAL
INFO    01:22:51 subscriber-3 sc_stage.py:293::process_batch_request_response | Node 3 marking batch ItemReq(Item(type:3, id:3-ATDGA998), qty:1) in-transit in local WAL
INFO    01:22:56 subscriber-3 sc_stage.py:293::process_batch_request_response | Node 3 marking batch ItemReq(Item(type:3, id:3-XNIGS106), qty:1) in-transit in local WAL
INFO    01:23:00 test ops runner-0 sc_stage.py:103::_attempt_inbound_recovery | Node 0 attempting post-crash recovery to verify status of batches in inbound WAL of size 2
INFO    01:23:00 test ops runner-0 sc_stage.py:108::_attempt_inbound_recovery | Node 0 found batch ItemReq(Item(type:0, id:0-FFLGZ803), qty:1) in state in-transit in inbound WAL
INFO    01:23:00 test ops runner-0 sc_stage.py:108::_attempt_inbound_recovery | Node 0 found batch ItemReq(Item(type:0, id:0-ZEWEE371), qty:1) in state in-transit in inbound WAL
INFO    01:23:00 test ops runner-0 sc_stage.py:130::_attempt_outbound_recovery | Node 0 attempting post-crash recovery to verify status of batches in outbound WAL of size 2
INFO    01:23:00 test ops runner-0 sc_stage.py:135::_attempt_outbound_recovery | Node 0 found batch ItemReq(Item(type:0, id:0-FFLGZ803), qty:1) in state delivered in outbound WAL
INFO    01:23:00 test ops runner-0 sc_stage.py:135::_attempt_outbound_recovery | Node 0 found batch ItemReq(Item(type:0, id:0-ZEWEE371), qty:1) in state in-transit in outbound WAL
INFO    01:23:00 subscriber-5 sc_stage.py:293::process_batch_request_response | Node 5 marking batch ItemReq(Item(type:0, id:0-ZEWEE371), qty:1) in-transit in local WAL
INFO    01:23:02 subscriber-1 sc_stage.py:293::process_batch_request_response | Node 1 marking batch ItemReq(Item(type:2, id:2-RMDBH885), qty:1) in-transit in local WAL
INFO    01:23:07 subscriber-1 sc_stage.py:293::process_batch_request_response | Node 1 marking batch ItemReq(Item(type:2, id:2-GWXNU514), qty:1) in-transit in local WAL
INFO    01:23:20 test ops runner-6 sc_stage.py:87::_attempt_log_recovery | Node 6's batch status WALs are empty. Skipping post-crash item and material status verification

Statistics:

,timestamp,node_id,metric_name,value
0,2020-06-07 01:23:46.848012,5,sent_messages,52.0
1,2020-06-07 01:23:00.849711,5,received_messages,50.0
2,2020-06-07 01:23:46.800203,5,total_manufacture_cycles,14.0
3,2020-06-07 01:23:46.807629,5,batches_produced,2.0
4,2020-06-07 01:23:46.813417,5,batches_consumed,2.0
5,2020-06-07 01:23:46.819269,5,unanswered_batch_requests,9.0
6,2020-06-07 01:23:46.825369,5,outbound_wal_size,2.0
7,2020-06-07 01:23:46.831316,5,inbound_wal_size,4.0
8,2020-06-07 01:23:46.840651,5,flow_queries,22.0
9,2020-06-07 01:23:46.854852,5,batches_requested,12.0
10,2020-06-07 01:23:46.863389,5,skipped_manufacture_cycles,12.0
11,2020-06-07 01:22:50.867954,5,batches_received,2.0
12,2020-06-07 01:22:55.812037,5,successful_manufacture_cycles,2.0
13,2020-06-07 01:22:56.657761,5,batches_delivered,1.0
chuanqichen commented 4 years ago

Great work! 👯‍♂️

Awesome!