Hello, I am considering a scenario that some of the SL nodes become offline during the training process. If the online SL nodes are more than min_peers, the SL training process can still continue.
However, I wonder how those offline nodes would be initialized when they come back online. Will the SL network consider the learnings of the offline nodes, to merge a new model? Or, will the SL network abandon the learnings of the offline nodes, and simply initialize them with merged models of online SL nodes in the latest batch?
Besides, when I try to simulate this scenario, I get an error called "SMLBB: Node enrolled cannot change ML context". Does SL support manually removing some epochs or batches from synchronization? If so, how can I implement this feature?
I really appreciate it if you provide me some hints. Thank you!
Hello, I am considering a scenario that some of the SL nodes become offline during the training process. If the online SL nodes are more than min_peers, the SL training process can still continue. However, I wonder how those offline nodes would be initialized when they come back online. Will the SL network consider the learnings of the offline nodes, to merge a new model? Or, will the SL network abandon the learnings of the offline nodes, and simply initialize them with merged models of online SL nodes in the latest batch? Besides, when I try to simulate this scenario, I get an error called "SMLBB: Node enrolled cannot change ML context". Does SL support manually removing some epochs or batches from synchronization? If so, how can I implement this feature? I really appreciate it if you provide me some hints. Thank you!