search

LabeliaLabs / distributed-learning-contributivity

Simulate collaborative ML scenarios, experiment multi-partner learning approaches and measure respective contributions of different datasets to model performance.
https://www.labelia.org
Apache License 2.0
56 stars 12 forks source link

Change log format #322

Closed arthurPignet closed 3 years ago

arthurPignet commented 3 years ago

I don't think we need the full track of which line of which function of which module triggers what log on every-day-utilisation

old log


2021-01-14 14:06:09.129 | INFO     | mplc.splitter:split:44 - ### Splitting data among partners:
2021-01-14 14:06:09.130 | INFO     | mplc.splitter:split:45 - Train data split:
2021-01-14 14:06:09.273 | INFO     | mplc.splitter:split:58 -    Partner #0: 4500 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.274 | INFO     | mplc.splitter:split:58 -    Partner #1: 4500 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.276 | INFO     | mplc.splitter:split:58 -    Partner #2: 4500 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.278 | INFO     | mplc.splitter:split:58 -    Partner #3: 4500 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.281 | INFO     | mplc.splitter:split:58 -    Partner #4: 4500 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.282 | INFO     | mplc.splitter:split:58 -    Partner #5: 4500 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.285 | INFO     | mplc.splitter:split:58 -    Partner #6: 4499 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.287 | INFO     | mplc.splitter:split:58 -    Partner #7: 4501 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.290 | INFO     | mplc.splitter:split:58 -    Partner #8: 4499 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.292 | INFO     | mplc.splitter:split:58 -    Partner #9: 4501 samples with labels [0 1 2 3 4 5 6 7 8 9]
2021-01-14 14:06:09.295 | INFO     | mplc.scenario:log_scenario_description:407 - Description of data scenario configured:
2021-01-14 14:06:09.297 | INFO     | mplc.scenario:log_scenario_description:408 -    Number of partners defined: 10
2021-01-14 14:06:09.298 | INFO     | mplc.scenario:log_scenario_description:409 -    Data distribution scenario chosen: Random samples split
2021-01-14 14:06:09.301 | INFO     | mplc.scenario:log_scenario_description:410 -    Multi-partner learning approach: fedavg
2021-01-14 14:06:09.302 | INFO     | mplc.scenario:log_scenario_description:411 -    Weighting option: data-volume
2021-01-14 14:06:09.302 | INFO     | mplc.scenario:log_scenario_description:412 -    Iterations parameters: 20 epochs > 1 mini-batches > 32 gradient updates per pass
2021-01-14 14:06:09.303 | INFO     | mplc.scenario:log_scenario_description:418 - Data loaded: cifar10
2021-01-14 14:06:09.306 | INFO     | mplc.scenario:log_scenario_description:423 -    45000 train data with 45000 labels
2021-01-14 14:06:09.307 | INFO     | mplc.scenario:log_scenario_description:426 -    5000 val data with 5000 labels
2021-01-14 14:06:09.307 | INFO     | mplc.scenario:log_scenario_description:429 -    10000 test data with 10000 labels
2021-01-14 14:06:09.307 | INFO     | __main__:__init__:31 - ## Preparation of model's training on partners with ids: ['#0', '#1', '#2', '#3', '#4', '#5', '#6', '#7', '#8', '#9']
2021-01-14 14:06:09.816 | INFO     | mplc.multi_partner_learning:init_model:130 - Init new model
2021-01-14 14:06:16.587 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 1 /20 - 7.0 s . loss: 2.05  -- accuracy: 0.24  -- val_loss: 2.18  -- val_accuracy: 0.22 
2021-01-14 14:06:23.079 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 2 /20 - 6.0 s . loss: 1.89  -- accuracy: 0.31  -- val_loss: 1.96  -- val_accuracy: 0.28 
2021-01-14 14:06:29.564 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 3 /20 - 6.0 s . loss: 1.92  -- accuracy: 0.32  -- val_loss: 1.98  -- val_accuracy: 0.34 
2021-01-14 14:06:35.944 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 4 /20 - 6.0 s . loss: 1.85  -- accuracy: 0.35  -- val_loss: 2.55  -- val_accuracy: 0.33 
2021-01-14 14:06:42.520 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 5 /20 - 7.0 s . loss: 1.85  -- accuracy: 0.36  -- val_loss: 2.74  -- val_accuracy: 0.29 
2021-01-14 14:06:48.788 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 6 /20 - 6.0 s . loss: 1.77  -- accuracy: 0.39  -- val_loss: 2.01  -- val_accuracy: 0.34 
2021-01-14 14:06:55.094 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 7 /20 - 6.0 s . loss: 1.63  -- accuracy: 0.41  -- val_loss: 1.99  -- val_accuracy: 0.32 
2021-01-14 14:07:01.306 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 8 /20 - 6.0 s . loss: 1.61  -- accuracy: 0.42  -- val_loss: 1.92  -- val_accuracy: 0.33 
2021-01-14 14:07:07.617 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 9 /20 - 6.0 s . loss: 1.55  -- accuracy: 0.44  -- val_loss: 1.62  -- val_accuracy: 0.4  
2021-01-14 14:07:13.849 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 10/20 - 6.0 s . loss: 1.49  -- accuracy: 0.46  -- val_loss: 1.49  -- val_accuracy: 0.45 
2021-01-14 14:07:20.303 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 11/20 - 6.0 s . loss: 1.44  -- accuracy: 0.48  -- val_loss: 1.44  -- val_accuracy: 0.48 
2021-01-14 14:07:26.502 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 12/20 - 6.0 s . loss: 1.4   -- accuracy: 0.5   -- val_loss: 1.42  -- val_accuracy: 0.49 
2021-01-14 14:07:32.827 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 13/20 - 6.0 s . loss: 1.37  -- accuracy: 0.51  -- val_loss: 1.4   -- val_accuracy: 0.49 
2021-01-14 14:07:39.063 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 14/20 - 6.0 s . loss: 1.33  -- accuracy: 0.53  -- val_loss: 1.38  -- val_accuracy: 0.5  
2021-01-14 14:07:45.367 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 15/20 - 6.0 s . loss: 1.3   -- accuracy: 0.54  -- val_loss: 1.33  -- val_accuracy: 0.52 
2021-01-14 14:07:51.717 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 16/20 - 6.0 s . loss: 1.26  -- accuracy: 0.55  -- val_loss: 1.26  -- val_accuracy: 0.56 
2021-01-14 14:07:58.175 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 17/20 - 6.0 s . loss: 1.23  -- accuracy: 0.56  -- val_loss: 1.22  -- val_accuracy: 0.57 
2021-01-14 14:08:04.518 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 18/20 - 6.0 s . loss: 1.2   -- accuracy: 0.57  -- val_loss: 1.19  -- val_accuracy: 0.58 
2021-01-14 14:08:10.916 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 19/20 - 6.0 s . loss: 1.17  -- accuracy: 0.58  -- val_loss: 1.17  -- val_accuracy: 0.59 
2021-01-14 14:08:17.175 | INFO     | __main__:log_epoch:83 - [FedGDOf] > Epoch 20/20 - 6.0 s . loss: 1.14  -- accuracy: 0.6   -- val_loss: 1.14  -- val_accuracy: 0.6  
2021-01-14 14:08:17.561 | INFO     | __main__:fit:149 - [FedGDOf] > Training 20 in 127.309 secondes. Tests scores: loss: 1.134 -- accuracy: 0.601

new log

2021-01-14 17:54:30 | INFO    | ### Splitting data among partners:
2021-01-14 17:54:30 | INFO    | Train data split:
2021-01-14 17:54:36 | INFO    |    Partner #0: 9000 samples with labels [8 9]
2021-01-14 17:54:36 | INFO    |    Partner #1: 9000 samples with labels [6 7]
2021-01-14 17:54:36 | INFO    |    Partner #2: 9000 samples with labels [4 5]
2021-01-14 17:54:36 | INFO    |    Partner #3: 9000 samples with labels [2 3]
2021-01-14 17:54:36 | INFO    |    Partner #4: 9000 samples with labels [0 1]
2021-01-14 17:54:36 | INFO    | Description of data scenario configured:
2021-01-14 17:54:36 | INFO    |    Number of partners defined: 5
2021-01-14 17:54:36 | INFO    |    Data distribution scenario chosen: Stratified samples split
2021-01-14 17:54:36 | INFO    |    Multi-partner learning approach: fedavg
2021-01-14 17:54:36 | INFO    |    Weighting option: data-volume
2021-01-14 17:54:36 | INFO    |    Iterations parameters: 20 epochs > 1 mini-batches > 32 gradient updates per pass
2021-01-14 17:54:36 | INFO    | Data loaded: cifar10
2021-01-14 17:54:36 | INFO    |    45000 train data with 45000 labels
2021-01-14 17:54:36 | INFO    |    5000 val data with 5000 labels
2021-01-14 17:54:36 | INFO    |    10000 test data with 10000 labels
2021-01-14 17:54:36 | INFO    | ## Preparation of model's training on partners with ids: ['#0', '#1', '#2', '#3', '#4']
2021-01-14 17:54:37 | INFO    | Init new model
2021-01-14 17:54:41 | INFO    | [FedGDOf] > Epoch 1 /20 - 5.0 s . loss: 0.78  -- accuracy: 0.57  -- val_loss: 2.31  -- val_accuracy: 0.11 
2021-01-14 17:54:46 | INFO    | [FedGDOf] > Epoch 2 /20 - 4.0 s . loss: 0.76  -- accuracy: 0.62  -- val_loss: 2.31  -- val_accuracy: 0.11 
2021-01-14 17:54:50 | INFO    | [FedGDOf] > Epoch 3 /20 - 4.0 s . loss: 0.63  -- accuracy: 0.69  -- val_loss: 2.39  -- val_accuracy: 0.16 
2021-01-14 17:54:54 | INFO    | [FedGDOf] > Epoch 4 /20 - 4.0 s . loss: 0.61  -- accuracy: 0.71  -- val_loss: 2.73  -- val_accuracy: 0.18 
2021-01-14 17:54:58 | INFO    | [FedGDOf] > Epoch 5 /20 - 4.0 s . loss: 0.6   -- accuracy: 0.72  -- val_loss: 2.31  -- val_accuracy: 0.22 
2021-01-14 17:55:02 | INFO    | [FedGDOf] > Epoch 6 /20 - 4.0 s . loss: 0.6   -- accuracy: 0.73  -- val_loss: 2.52  -- val_accuracy: 0.19 
2021-01-14 17:55:06 | INFO    | [FedGDOf] > Epoch 7 /20 - 4.0 s . loss: 0.58  -- accuracy: 0.74  -- val_loss: 2.21  -- val_accuracy: 0.22 
2021-01-14 17:55:10 | INFO    | [FedGDOf] > Epoch 8 /20 - 4.0 s . loss: 0.53  -- accuracy: 0.76  -- val_loss: 2.12  -- val_accuracy: 0.22 
2021-01-14 17:55:14 | INFO    | [FedGDOf] > Epoch 9 /20 - 4.0 s . loss: 0.52  -- accuracy: 0.77  -- val_loss: 2.1   -- val_accuracy: 0.23 
2021-01-14 17:55:18 | INFO    | [FedGDOf] > Epoch 10/20 - 4.0 s . loss: 0.51  -- accuracy: 0.78  -- val_loss: 2.08  -- val_accuracy: 0.23 
2021-01-14 17:55:22 | INFO    | [FedGDOf] > Epoch 11/20 - 4.0 s . loss: 0.5   -- accuracy: 0.78  -- val_loss: 2.06  -- val_accuracy: 0.24 
2021-01-14 17:55:26 | INFO    | [FedGDOf] > Epoch 12/20 - 4.0 s . loss: 0.48  -- accuracy: 0.79  -- val_loss: 2.03  -- val_accuracy: 0.27 
2021-01-14 17:55:30 | INFO    | [FedGDOf] > Epoch 13/20 - 4.0 s . loss: 0.46  -- accuracy: 0.8   -- val_loss: 1.97  -- val_accuracy: 0.31 
2021-01-14 17:55:34 | INFO    | [FedGDOf] > Epoch 14/20 - 4.0 s . loss: 0.44  -- accuracy: 0.81  -- val_loss: 1.97  -- val_accuracy: 0.31 
2021-01-14 17:55:38 | INFO    | [FedGDOf] > Epoch 15/20 - 4.0 s . loss: 0.44  -- accuracy: 0.81  -- val_loss: 2.02  -- val_accuracy: 0.28 
2021-01-14 17:55:43 | INFO    | [FedGDOf] > Epoch 16/20 - 4.0 s . loss: 0.42  -- accuracy: 0.82  -- val_loss: 2.05  -- val_accuracy: 0.24 
2021-01-14 17:55:47 | INFO    | [FedGDOf] > Epoch 17/20 - 4.0 s . loss: 0.42  -- accuracy: 0.82  -- val_loss: 2.03  -- val_accuracy: 0.25 
2021-01-14 17:55:51 | INFO    | [FedGDOf] > Epoch 18/20 - 4.0 s . loss: 0.4   -- accuracy: 0.83  -- val_loss: 1.95  -- val_accuracy: 0.29 
2021-01-14 17:55:55 | INFO    | [FedGDOf] > Epoch 19/20 - 4.0 s . loss: 0.4   -- accuracy: 0.83  -- val_loss: 1.91  -- val_accuracy: 0.32 
2021-01-14 17:55:59 | INFO    | [FedGDOf] > Epoch 20/20 - 4.0 s . loss: 0.4   -- accuracy: 0.83  -- val_loss: 1.92  -- val_accuracy: 0.32 
2021-01-14 17:55:59 | INFO    | [FedGDOf] > Training 20 in 82.235 secondes. Tests scores: loss: 1.912 -- accuracy: 0.318

By the way the scenario logs are quite a mess, sometime with ##, sometime without. I create an issue about that