search

qthequartermasterman / hypothesis-torch

Hypothesis strategies for various Pytorch structures (including tensors and modules).
https://hypothesis-torch.readthedocs.io/en/latest/
MIT License
6 stars 2 forks source link

Feature Request: Add support for generated distributed tensors #125

Open qthequartermasterman opened 1 month ago

qthequartermasterman commented 1 month ago

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

https://pytorch.org/docs/stable/distributed.tensor.parallel.html

There is a distributed tensor type (DTensor) that is not compatible with all torch operations. It would be nice to allow tensor_strategy to create Dtensors.

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

qthequartermasterman commented 1 month ago

This is how PyTorch is currently testing DTensors, courtesy of @Nan2018.

https://github.com/pytorch/pytorch/blob/main/torch/testing/_internal/common_distributed.py#L521

qthequartermasterman commented 5 days ago

Looks like they've moved some tests around. I think this is the new location:

https://github.com/pytorch/pytorch/blob/071d48c56e6451f95a87507cddd68e836428a68e/torch/testing/_internal/distributed/_tensor/common_dtensor.py#L297