Evaluation done on 2 clients

minaremeli / MOON

MOON reproducibility study

0 stars 0 forks source link

Closed minaremeli closed 2 years ago

minaremeli commented 2 years ago

In each round we fit on fraction * num_clients.

In each round we also evaluate on client's local dataset. Currently evaluation is calculated on a fix number of 2 clients.

Expected behaviour is not clear, and there are multiple options:

always evaluate on the full set of clients
evaluate on fraction * num_clients (random sampling)
evaluate on fraction * num_clients (SAME clients as before, when we fit the model)

@VasundharaAgarwal what do you think?

minaremeli commented 2 years ago

Btw the bug is here: fraction_eval is not set, and thus defaults to 0.1 all the time.

VasundharaAgarwal commented 2 years ago

Ah sorry, fraction_eval should be set to 1.0 then. We would want to evaluate on all the clients after every round. Thanks for catching this!

minaremeli commented 2 years ago

Thanks for the prompt response!