-
When we try to execute terraform plan and more than 100 topics will be created, the process errors out claiming we have exceeded a limit of 100 partitions. In the example below, we were trying to crea…
-
Some things should eventually have parallelization options:
- GRAPE
- Over Distributions
- Over initial guesses
- Over the time sliced unitary list
- QSim
- Over Distribitions (should be easy)…
-
If scanning many projects, I suggest using a single thread/process per project instead. Easier to reason about.
But if scanning a single project, each analysis phase can be performed across multipl…
-
# Motivation
On Hopper, [efficient gemm requires warp-specialization](https://github.com/NVIDIA/cutlass/blob/main/media/docs/efficient_gemm.md#warp-specialization), which is not currently supported…
-
```
assert not args.model_parallel.fp16, \
"Expert parallelism is not supported with fp16 training."
```
from https://github.com/NVIDIA/Megatron-LM/blob/db3a3f79d1cda60ea4b3db0ceffcf…
-
`diagrams-haddock` has some embarrassing parallelism going on. We should add a -j flag that turns on multithreading.
-
Implement distributed attention in LightSeq, Colossal-AI, or DeepSpeed's SP.... We have not decided which one yet.
```
from pipegoose.nn.sequence_parallel.attention import DistributedAttention
…
-
If we enable expert parallelism, there will be two optimizers for dense parameters and expert parameters. When we call `optimizer.step() ` the two optimizers perform grad-norm for their own parameters…
-
This issue is complete when the code runs with OpenMP parallelism.
-
# Parallelism in SEMPR
## Intro
As discussed before, we want some kind of parallelism in sempr. The big questions are:
1. What do we want to compute in parallel?
2. How do we achieve that? What do…