raokrutarth / manufacturing-os

A POC distributed operating system for geo-seperated automotive manufacturing
Creative Commons Zero v1.0 Universal
1 stars 0 forks source link

Add expriment running and metric analysis automation #136

Open raokrutarth opened 4 years ago

raokrutarth commented 4 years ago

run with

make small-no-fail
raokrutarth commented 4 years ago

Tested with latest master (i.e. @chuanqichen 's changes.) evaluation with 5x2 below:

[14:41:43] Metric-wise breakdown:

                                       value                                   
                                         min    max   mean median   std     var
metric_name                                                                    
BatchDeliveryConfirmResponse_received   2.00  36.00  16.00  13.00 16.47  271.33
BatchDeliveryConfirmResponse_sent       5.00  35.00  18.25  16.50 12.74  162.25
BatchRequest_received                  12.00 103.00  61.25  65.00 39.89 1591.58
BatchRequest_sent                      48.00  93.00  68.75  67.00 18.64  347.58
BatchSentResponse_received              5.00  37.00  19.25  17.50 13.38  178.92
BatchSentResponse_sent                  4.00  48.00  26.75  27.50 19.00  360.92
BatchUnavailableResponse_received      24.00  37.00  29.67  28.00  6.66   44.33
BatchUnavailableResponse_sent           8.00  68.00  46.00  62.00 33.05 1092.00
HeartbeatReq_received                  17.00 112.00  96.50 112.00 33.57 1126.86
HeartbeatReq_sent                      15.00  16.00  15.75  16.00  0.46    0.21
HeartbeatResp_received                 13.00  95.00  81.38  94.00 28.61  818.55
HeartbeatResp_sent                     17.00 112.00  96.62 112.00 33.63 1131.12
InformLeaderOfDeathReq_received         1.00   1.00   1.00   1.00   NaN     NaN
InformLeaderOfDeathReq_sent             6.00  63.00  14.88   9.00 19.48  379.55
WaitingForBatchResponse_received        2.00  37.00  16.75  14.00 17.17  294.92
WaitingForBatchResponse_sent            5.00  37.00  19.25  17.50 13.38  178.92
batch_unavailable_messages_received    24.00  37.00  29.67  28.00  6.66   44.33
batch_unavailable_messages_sent         0.00  68.00  17.25   0.00 29.64  878.79
batches_consumed_total                  0.00  35.00   9.12   2.00 12.93  167.27
batches_delivered                       0.00  36.00   8.00   1.00 13.76  189.43
batches_produced_total                  2.00  85.00  51.12  59.00 36.80 1354.12
batches_received                        5.00  36.00  18.75  17.00 13.05  170.25
batches_requested                      48.00  93.00  68.75  67.00 18.64  347.58
bootstrap_all_paths_time_sec            0.00   0.00   0.00   0.00  0.00    0.00
empty_inbound_inventory_occurrences     0.00  93.00  34.38  24.00 38.72 1499.41
empty_outbound_inventory_occurrences    0.00  68.00  17.25   0.00 29.64  878.79
failed_flow_queries                     0.00   0.00   0.00   0.00  0.00    0.00
flow_queries                           84.00 186.00 115.00  91.00 41.22 1699.14
full_flow_bootstrap_time_sec            0.01   0.01   0.01   0.01  0.00    0.00
inbound_wal_size                        0.00  71.00  22.75  12.00 26.98  727.93
nodes_determined_crashed                6.00  63.00  14.88   9.00 19.48  379.55
outbound_material_buildup               0.00  85.00  37.62  23.00 40.38 1630.27
outbound_wal_size                       2.00  85.00  50.88  58.50 36.55 1335.55
received_messages                      60.00 345.00 245.88 241.00 96.57 9325.27
sent_messages                         134.00 330.00 211.00 191.50 80.92 6547.43
skipped_manufacture_cycles              0.00  81.00  32.50  24.00 35.90 1288.57
successful_manufacture_cycles           2.00  85.00  51.25  59.50 36.92 1363.36
total_manufacture_cycles               84.00  86.00  84.75  84.00  1.04    1.07
unanswered_batch_requests_current       0.00  64.00  13.62   6.00 21.73  471.98
wal_ghost_inbound_batches               0.00   0.00   0.00   0.00  0.00    0.00
wal_ghost_outbound_batches              0.00   0.00   0.00   0.00  0.00    0.00
wal_recovered_inbound_batches           0.00   0.00   0.00   0.00  0.00    0.00
wal_recovered_outbound_batches          0.00   0.00   0.00   0.00  0.00    0.00

===

zee@zee-Z620:~/sharp/manufacturing-os$ 
raokrutarth commented 4 years ago

NVM. Seeing crashes later:

image