search

zhao-zilong / Tabula

Official git for "TabuLa: Harnessing Language Models for Tabular Data Synthesis"
27 stars 9 forks source link

How to train model across multiple GPUs? #12

Open HatedFate opened 4 days ago

HatedFate commented 4 days ago

I am still very new to LLMs. I have access to a large amount of GPUs, and I would like to train this model across multiple GPUs ( though, I am not sure whether this is necessary/overkill ). Previously, I used DistributedDataParallel for parallelization, but I am not sure how to integrate this into the trainer.

zhao-zilong commented 4 days ago

Hi @HatedFate I think you do not need to do anything, HuggingFace already did that for you. You should already trained the model on multi-GPU

HatedFate commented 4 days ago

Hi @HatedFate I think you do not need to do anything, HuggingFace already did that for you. You should already trained the model on multi-GPU

So I can simply run it the same way it is done in the Jupyter Notebook, right? Do I have to specified how many GPUs I am using or will it default to use all the GPUs I allocated to it?

zhao-zilong commented 3 days ago

@HatedFate yeah, I think so. I will allocate all available sources by default.