Since the dataset on each host is unbalanced, should find out a better way to do sync since on host3 with the smallest dataset always finish the epochs faster and will loop in the local small dataset. Causing overfitting.
Approaches could be:
Try different sync intervals according to different size of ds
try larger epochs for hosts with smaller ds
try different weight learning for hosts. Host with smaller ds should weigh less(gain less weight) during local training
put on_batch_end to on_epoch_end
it would be better that host would be able to wait for others to reach epoch end then do sync(currently I think it would initiate the sync when it reaches the sync frequency with
Since the dataset on each host is unbalanced, should find out a better way to do sync since on host3 with the smallest dataset always finish the epochs faster and will loop in the local small dataset. Causing overfitting. Approaches could be: