Great Work !
I found that the code works well on multiple gpu training but only for the data parallelism.
However, it is hard to train a model with the classes more than 10 million or even more. In that case,
the model parallelism should solve the problem. I was doing the implementations but the diam softmax may seriously interfere the model parallelism. Could you give the solutions or any idea ?
Great Work ! I found that the code works well on multiple gpu training but only for the data parallelism. However, it is hard to train a model with the classes more than 10 million or even more. In that case, the model parallelism should solve the problem. I was doing the implementations but the diam softmax may seriously interfere the model parallelism. Could you give the solutions or any idea ?