Features: Data parallel for more flexibility and more efficiency

seoulsky-field / CXRAIL-dev

CXRAIL-dev

MIT License

7 stars 0 forks source link

Features: Data parallel for more flexibility and more efficiency #97

Open seoulsky-field opened 1 year ago

seoulsky-field commented 1 year ago

What

When I do experiments using multi-gpu, I was faced of training slow down, imbalanced data assignment, some problems of multi processing.

Why

When we use multiple gpu, we should correctly use PyTorch programming in multi processing. (For server and us) However, because of time consuming, this is a future work.

How

[ ] Use DistributedDataParallel
[ ] Use DistributedSampler
reference: https://kongsberg.tistory.com/7

kdg1993 commented 1 year ago

Thanks for considering the multi-GPU issue!

As far as I know, the DistributedDataParallel is a bit complex compared to the DataParallel (multi-processing vs multi-threading), So I think the official PyTorch document is worth reading once.

I think you already know this well but I leave the similar issue #54 just in case 😄 👍