Data parallelism or tensor parallelism? How can i know that and is there a chance i can shift in between these too? - Githubissues

huggingface / optimum-habana

Easy and lightning fast training of 🤗 Transformers on Habana Gaudi processor (HPU)

Apache License 2.0

145 stars 181 forks source link

Data parallelism or tensor parallelism? How can i know that and is there a chance i can shift in between these too? #1242

Open venkycreator opened 1 month ago

venkycreator commented 1 month ago

System Info

in the run_generation.py (for text generation) How can i know that what kind of parallelism it is like data or tensor? and is there a chance to shift in between these two?

Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

I wanted to know in order to rum the models parallel using deepspeed

Expected behavior

wanted to know what kind of parallelism it supports

regisss commented 1 month ago

@venkycreator This example doesn't offer any data parallelism. It could be added wrapping the model with PyTorch DistributedDataParallel.

Tensor parallelism is possible either using DeepSpeed or adding the argument --parallel_strategy tp.