-
Dear Dr. Wilson,
I plan to implement and test a couple of changes in my fork. It would be great to hear your opinions on these changes before I proceed.
1. **Change in Parallel Backend**: I prop…
hhp94 updated
2 months ago
-
`--num-gpus` is implemented by [sharding each expert layer across GPUs](https://github.com/open-compass/MixtralKit/blob/38bbb5524ee6dcecd2d4724b06179c6783019db4/mixtralkit/layers/moe.py#L27), i.e. exp…
-
- torch::torch_set_num_threads(n)
- torch::torch_set_num_interop_threads(n)
missing
-
If we enable expert parallelism, there will be two optimizers for dense parameters and expert parameters. When we call `optimizer.step() ` the two optimizers perform grad-norm for their own parameters…
-
### Details
Currently, ABACUS enables molecular dynamics calculations with DeePMD-kit. However, the MPI parallelism does not work, which affects the performance.
The MPI parallelism should be supp…
-
Multithreaded calls to rand serialize. Each thread should have their own RNG.
-
**Is your feature request related to a problem? Please describe.**
This is a meta issue to track progress of parallelism in moirai. The current approach does not scale vertically or horizontally.
…
-
I ran a conversion of a 45 gig plink file as described over in https://github.com/pystatgen/sgkit/issues/48
Overall, it worked great, and I was super impressed by how smooth the process was. Howeve…
-
How does RDataTracker work with a package like snow or other parallelism mechanisms?
-
torchxla spmd whether expert parallelism is supported?
If it is a moe model, how should it be computed in xla?
## ❓ Questions and Help