search

helmholtz-analytics / heat

Distributed tensors and Machine Learning framework with GPU and MPI acceleration in Python
https://heat.readthedocs.io/
MIT License
212 stars 53 forks source link

Feature: DataParallel: async SGD #586

Open coquelin77 opened 4 years ago

coquelin77 commented 4 years ago

asynchronous SGD also has many options, 1 option which looks promising now is a ring approach

coquelin77 commented 3 years ago

there is a version of this implemented in heat now however it is based on nonblocking hooks but it still does traditional synchronization SGD. that is not traditional ASGD. there is another parallelization method in DASO (#728 ) but that is a different optimization method.

ClaudiaComito commented 2 years ago

What should be done here @coquelin77 ?

mrfh92 commented 1 year ago

In principle of interest, if we want to extend/improve NN support. Could be a good student project if a sufficiently detailed description could be added.

Reviewed within #1109