search

LabeliaLabs / distributed-learning-contributivity

Simulate collaborative ML scenarios, experiment multi-partner learning approaches and measure respective contributions of different datasets to model performance.
https://www.labelia.org
Apache License 2.0
56 stars 12 forks source link

Mpl refacto #255

Closed arthurPignet closed 3 years ago

arthurPignet commented 3 years ago

The final objective is to refacto the mpl methods into classes. The use of the library will be clearer, and add a new method will be easier.

Don't hesitate to comment ! This is a major refacto, with many different possible solutions.

natct10 commented 3 years ago

LGTM but tests are failing on mnist & contrib

natct10 commented 3 years ago

My bad, misclick!

natct10 commented 3 years ago

The mnist test is failing as the min score is 0.92. Do you want what's happening? Should we lower the test threshold to 0.92 instead of 0.95?

        df = Test_EndToEndTest.get_latest_dataframe("*end_to_end_test*")
        # Extract score
        min_test_score = df["mpl_test_score"].min()
>       assert min_test_score > 0.95
E       assert 0.9215999841690063 > 0.95
tests/end_to_end_tests.py:43: AssertionError
natct10 commented 3 years ago

I have the same error than travis:

ERROR    | __main__:<module>:115 - An error has been caught in function '<module>', process 'MainProcess' (231709), thread 'MainThread' (139951218480960):
Traceback (most recent call last):

> File "main.py", line 115, in <module>
    main()
    └ <function main at 0x7f48d1e54e50>

  File "main.py", line 78, in main
    current_scenario.run()
    │                └ <function Scenario.run at 0x7f485d45c040>
    └ <mplc.scenario.Scenario object at 0x7f485cd08940>

  File "/home/zen/dev/distributed-learning-contributivity/mplc/scenario.py", line 849, in run
    self.mpl.fit()
    │    │   └ <function MultiPartnerLearning.fit at 0x7f4866d390d0>
    │    └ <mplc.multi_partner_learning.FederatedAverageLearning object at 0x7f4855ca6e20>
    └ <mplc.scenario.Scenario object at 0x7f485cd08940>

  File "/home/zen/dev/distributed-learning-contributivity/mplc/multi_partner_learning.py", line 207, in fit
    self.save_final_model()
    │    └ <function MultiPartnerLearning.save_final_model at 0x7f4866d36d30>
    └ <mplc.multi_partner_learning.FederatedAverageLearning object at 0x7f4855ca6e20>

  File "/home/zen/dev/distributed-learning-contributivity/mplc/multi_partner_learning.py", line 128, in save_final_model
    model_to_save.save_weights(os.path.join(model_folder, self.dataset_name + '_final_weights.h5'))
    │             │            │  │    │    │             │    └ 'mnist'
    │             │            │  │    │    │             └ <mplc.multi_partner_learning.FederatedAverageLearning object at 0x7f4855ca6e20>
    │             │            │  │    │    └ '/home/zen/dev/distributed-learning-contributivity/experiments/end_to_end_test_mnist_2020-11-03_10h36/scenario_1_repeat_1_202...
    │             │            │  │    └ <function join at 0x7f48ee3875e0>
    │             │            │  └ <module 'posixpath' from '/usr/lib/python3.8/posixpath.py'>
    │             │            └ <module 'os' from '/usr/lib/python3.8/os.py'>
    │             └ <function Network.save_weights at 0x7f4866e7fc10>
    └ <keras.engine.sequential.Sequential object at 0x7f48442a43a0>

  File "/home/zen/.local/lib/python3.8/site-packages/keras/engine/saving.py", line 449, in save_wrapper
    save_function(obj, filepath, overwrite, *args, **kwargs)
    │             │    │         │           │       └ {}
    │             │    │         │           └ ()
    │             │    │         └ True
    │             │    └ '/home/zen/dev/distributed-learning-contributivity/experiments/end_to_end_test_mnist_2020-11-03_10h36/scenario_1_repeat_1_202...
    │             └ <keras.engine.sequential.Sequential object at 0x7f48442a43a0>
    └ <function Network.save_weights at 0x7f4866e7fb80>
  File "/home/zen/.local/lib/python3.8/site-packages/keras/engine/network.py", line 1184, in save_weights
    saving.save_weights_to_hdf5_group(f, self.layers)
    │      │                          │  │    └ <property object at 0x7f4866d6af40>
    │      │                          │  └ <keras.engine.sequential.Sequential object at 0x7f48442a43a0>
    │      │                          └ <Closed HDF5 file>
    │      └ <function save_weights_to_hdf5_group at 0x7f4866e7e940>
    └ <module 'keras.engine.saving' from '/home/zen/.local/lib/python3.8/site-packages/keras/engine/saving.py'>
  File "/home/zen/.local/lib/python3.8/site-packages/keras/engine/saving.py", line 760, in save_weights_to_hdf5_group
    param_dset = g.create_dataset(name, val.shape,
                 │ │              │     │   └ <attribute 'shape' of 'numpy.ndarray' objects>
                 │ │              │     └ array([[[[-0.03707604, -0.03573737,  0.13777296,  0.17350416,
                 │ │              │                  0.02721884,  0.07960887,  0.01194842,  0.05688735,
                 │ │              │        ...
                 │ │              └ b'conv2d_83/kernel:0'
                 │ └ <function Group.create_dataset at 0x7f48c1592310>
                 └ <Closed HDF5 group>
  File "/home/zen/.local/lib/python3.8/site-packages/h5py/_hl/group.py", line 143, in create_dataset
    if '/' in name:
              └ b'conv2d_83/kernel:0'

TypeError: a bytes-like object is required, not 'str'
natct10 commented 3 years ago

=> h5py==2.10.0