-
In my understanding, in pretrain code, it broadcasts the data from tp rank 0 to the rest tp rank gpus.
However, if i activate the option `train_valid_test_datasets_provider.is_distributed = True` wh…
-
Found in code evaluation with CP2K. Code hung with latest ifx compiler. Caused by nested OpenMP regions that doesn't allow by standard. Code path contain multiple source files, so only short snippets …
-
I believe that a useful feature would be to implement a wrapper for the pytorch distributed data parallel layer.
My personal motivation for this is to be able to use things like synchronized batch…
-
### Anything you want to discuss about vllm.
# Issue
I have been experimenting on CUDAGraph captured generation with my own transformer model implementation, using [custom all-reduce](https://github…
-
## 🐛 Bug
I tried to use multi-gpu capability in `C++`, but when I write `torch::nn::parallel::data_parallel`, I am getting `No member named 'parallel' in namespace 'torch::nn'` error.
## To Repro…
-
The following snippet should produce a figure where the annotation is parallel to the line, however if the figure size changes the rotation of the annotation doesn't change.
```python
import plotly.…
-
PyTorch supports "num_workers" on data loaders - add support for similar parallel data loading
```python
train_loader = DataLoader(dataset_train, batch_size=opt.batch_size, shuffle=True, num_w…
dsyme updated
2 years ago
-
After #20910, data parallel implementation still needs the following improvements:
- [ ] Fix C++ data parallel for BN
- [ ] Make sure C++ data parallel work for double backward
- [ ] Move `reduce…
-
In many cases, the full power of a tasking model is not needed, and people can safely perform parallel computations using simple parallel primitives inspired by functional languages such as map, filte…
-
### API Spec link
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/preview/2024-09-01-preview/inference.json
### API Spec vers…