issue description: We observed inconsistency in the final models trained by Swarm Learning. We have two nodes involved in swarm learning. However whether we are picking the last or best checkpoint for prediction the results are significantly different from each other.
Issue description
Python scripts used to reproduce this problem: base_model.txt main.txt
Swarm Learning Version:
2.2.0
OS and ML Platform
Quick Checklist: Respond [Yes/No]
Additional notes
NOTE: Create an archive with supporting artifacts and attach to issue, whenever applicable.