Fairscale to tensor.parallel upgrade for reference_impl of model ?

meta-llama / llama-models

Utilities intended for use with Llama models.

Other

4.65k stars 808 forks source link

Fairscale to tensor.parallel upgrade for reference_impl of model ? #127

Open whatdhack opened 2 months ago

whatdhack commented 2 months ago

Is there a pure PyTorch implementation using torch.distributed.tensor.parallel instead of fairscale.nn.model_parallel ? Fairscale package looks a bit old with not much activity lately. Also, it will be good to have a list of other known implementations - pure PyTorch or not.

ashwinb commented 1 month ago

@whatdhack thanks for the nudge. We will prioritize getting a pure pytorch reference implementation.