myleott
commented
4 years ago Several things:
- Is your vocabulary size quite large? The model has >2B parameters, which I suspect is partly caused by a large embedding table.
- You have
--tokens-per-sample 3072, which is too large of a context size for a model with this many parameters. Try 1024 or even smaller (512 could be fine). - You can use
--memory-efficient-fp16instead of--fp16. This is a slightly more aggressive version of mixed precision training which will save memory, but typically requires a large batch size and may produce slightly worse perplexities in the end.
Another idea is to use model parallel training. We support this here: https://github.com/pytorch/fairseq/tree/master/examples/megatron_11b#example-training-command-model-parallel
yuan-commits
stale[bot]
❓ Questions and Help
What is your question?
I am training a GPT2-big model on a large dataset (6600M words) on
NVIDIA V100 32G * 8. It shows OOM error when training. Any methods for optimize GPU memory and fix OOM?In addition, is there any instruction or demo training script for training gpt2-large besides the LM readme?
Code
Here is my training script, following the [adaptive LM readme](https://github.com/pytorch/fairseq/blob/master/examples/language_model/README.adaptive_inputs.md). Note that I enable fp16 option. ``` bash # max-tokens 2k fairseq-train --task language_modeling \ "$DATA_DIR" \ --save-dir "$SAVE_DIR" \ --arch transformer_lm_gpt2_big \ --optimizer nag --clip-norm 0.1 \ --lr 0.0001 --lr-scheduler cosine --max-lr 1.0 \ --t-mult 2 --lr-period-updates 270000 --lr-shrink 0.75 \ --warmup-updates 16000 --warmup-init-lr 1e-07 \ --max-tokens 2048 --update-freq 3 \ --tokens-per-sample 3072 --sample-break-mode eos --seed 1 \ --skip-invalid-size-inputs-valid-test --ddp-backend=no_c10d \ --min-lr 1e-09 \ --fp16 \ --save-interval-updates 2000 \ --keep-interval-updates 1 # max-tokens 64 fairseq-train --task language_modeling \ "$DATA_DIR" \ --save-dir "$SAVE_DIR" \ --arch transformer_lm_gpt2_big \ --optimizer nag --clip-norm 0.1 \ --lr 0.0001 --lr-scheduler cosine --max-lr 1.0 \ --t-mult 2 --lr-period-updates 270000 --lr-shrink 0.75 \ --warmup-updates 16000 --warmup-init-lr 1e-07 \ --max-tokens 64 --update-freq 96 \ --tokens-per-sample 3072 --sample-break-mode eos --seed 1 \ --skip-invalid-size-inputs-valid-test --ddp-backend=no_c10d \ --min-lr 1e-09 \ --fp16 \ --save-interval-updates 2000 \ --keep-interval-updates 1 ``` #### What have you tried? - I have train a transformer_lm_gpt2_small model with the same optimizing strategy, it works well. (not using adam with rvs_sqrt because it always show overflow under fp16) - Reduce the max-token even to 64 as well as increasing update_freq also results in OOM error. #### What's your environment? - fairseq Version (e.g., 1.0 or master): former master (commit 775122950d145382146e9120308432a9faf9a9b8) - PyTorch Version (e.g., 1.0) 1.4.0 - OS (e.g., Linux): Ubuntu 16.04.6 LTS - How you installed fairseq (`pip`, source): editable mode by using: ``` bash git clone https://github.com/pytorch/fairseq cd fairseq pip install --editable . ``` - Build command you used (if compiling from source): None - Python version: 3.7 - CUDA/cuDNN version: CUDA runtime version: 10.0.130 cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5 - GPU models and configuration: V100 32G * 8 - Any other relevant information: