search

facebookresearch / fairseq

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
MIT License
30.43k stars 6.4k forks source link

how to enable every sample in each GPU is from the same language during multilingual NMT training with multi-gpus? #4084

Closed ghchen18 closed 2 years ago

ghchen18 commented 2 years ago

🚀 Feature Request

During multilingual multi-gpu training, every batch is randomly sampled and may be not from the same language. How to enable all sentences in each batch of each GPU has the same language ID?

Motivation

The feature is related to contrastive learning which requires a large batch size. The large batch is enabled by all-gather operation after calculate the text feature, which is hopefully from the sample language.

Pitch

During a certain train step, all sentences in the batch of each GPU have the same source/target language ID during multilingual NMT training.

Alternatives

NA

Additional context

NA

Thanks for your time and help.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale. If this issue is still affecting you, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!

stale[bot] commented 2 years ago

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue with up-to-date information. Thank you!