-
### 🚀 The feature, motivation and pitch
# RFC: PyTorch DistributedTensor
We have been developing a DistributedTensor (a.k.a DTensor) concept under the [pytorch/tau](https://github.com/pytorch/ta…
-
This is to unblock @cowanmeg and @samnordmann 's distributed matmul experiments.
I'll start with the tensor parallelism proposed by [the original Megatron-LM paper](https://arxiv.org/pdf/1909.0805…
-
I didn't see any documentation that mentions that.
-
In Megatron, I find that the check for `tp_comm_overlap` and `sequence_parallel`。
```
if args.tp_comm_overlap:
assert args.sequence_parallel == True, 'Tensor parallel communicatio…
-
llava multimodel would be huge to be supported for aws neuron chips
https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf
This in particular is trending
I'm not sure if this is the correct…
-
I am trying to run examples/pretrain_vision_classify.sh. I am wondering if tensor parallelism and pipeline parallelism are supported for vision models ? In other words, can I use tensor-model-parallel…
-
### Your current environment
```text
The output of `python collect_env.py`
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" f…
-
**Is your feature request related to a problem? Please describe.**
Would be good to remove the megatron tensor parallelism code from NeoX, and [OSLO](https://github.com/tunib-ai/oslo) currently has s…
-
Hi!, Thank you for this amazing library, however it is not clear by documentation if it supports matrix/tensor multiplication..
Does it employ also thread parallelism? (Parallel.For, in addition to…
-
Dear Author,
Your contribution is critical for the open-source community. The distributed-llama repo has implemented tensor parallelism from scratch. And the result is amazingly significant. Howeve…