EricHallahan
commented
2 years ago I created a new Python 3.8.0 virtual env, installed torch 1.8.2+cu111, installed the latest NVIDIA apex, then installed all the requirements in each file within the requirements directory of GPT-NeoX.
I notice that you do not explicitly mention DeepSpeed or DeeperSpeed. As GPT-NeoX relies on DeeperSpeed for ZeRO and the rest of its features for distributed training, it is an important piece of the puzzle. DeeperSpeed should be installed from ./requirements/requirements.txt, so I would verify that install completes correctly and test ZeRO 2 on some other code so we can be sure this is a GPT-NeoX problem rather than a DeeperSpeed problem.
Starting on a host with NVIDIA driver 460.91.03, I ran a new docker container using image
nvidia/cuda:11.1.1-devel-ubuntu18.04.
I would strongly suggest using our prebuilt images or building your own image from the included Dockerfile—all of the critical dependencies will be preinstalled and ready to use. It is a lot easier to use our verified environment than to install the dependencies from scratch yourself.
pwstegman
StellaAthena
mallorbc
Describe the bug
When starting with
small.yml, then changing ZeRO to 2 andcpu_offloadtotrue, I get the following error:Similar to https://github.com/EleutherAI/gpt-neox/issues/171.
Full Traceback
NeoXArgs.from_ymls() ['configs/small.yml', 'configs/local_setup_small.yml'] INFO:root:NeoXArgs.calculate_derived() Total number of GPUs determined to be: 4 -------------------- arguments -------------------- attention_config ................ ['global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global', 'global']updated attention_dropout ............... 0.0.........................updated batch_size ...................... 4...........................updated checkpoint_activations .......... True........................updated clip_grad ....................... 1.0.........................updated config_files .................... {'small.yml': '# GPT-2 pretraining setup\n{\n # parallelism settings ( you will want to change these based on your cluster setup, ideally scheduling pipeline stages\n # across the node boundaries )\n "pipe-parallel-size": 1,\n "model-parallel-size": 1,\n\n # model settings\n "num-layers": 12,\n "hidden-size": 768,\n "num-attention-heads": 12,\n "seq-length": 2048,\n "max-position-embeddings": 2048,\n "norm": "layernorm",\n "pos-emb": "rotary",\n "no-weight-tying": true,\n\n # these should provide some speedup but takes a while to build, set to true if desired\n "scaled-upper-triang-masked-softmax-fusion": false,\n "bias-gelu-fusion": false,\n\n\n # optimizer settings\n "optimizer": {\n "type": "Adam",\n "params": {\n "lr": 0.0006,\n "betas": [0.9, 0.999],\n "eps": 1.0e-8,\n }\n },\n "zero_optimization": {\n "stage": 2,\n "allgather_partitions": True,\n "allgather_bucket_size": 500000000,\n "overlap_comm": True,\n "reduce_scatter": True,\n "reduce_bucket_size": 500000000,\n "contiguous_gradients": True,\n "cpu_offload": True\n },\n\n # batch / data settings\n "train_micro_batch_size_per_gpu": 4,\n "data-impl": "mmap",\n "split": "949,50,1",\n\n # activation checkpointing\n "checkpoint-activations": true,\n "checkpoint-num-layers": 1,\n "partition-activations": true,\n "synchronize-each-layer": true,\n\n # regularization\n "gradient_clipping": 1.0,\n "weight-decay": 0.0,\n "hidden-dropout": 0.0,\n "attention-dropout": 0.0,\n\n # precision settings\n "fp16": { \n "enabled": true,\n "loss_scale": 0,\n "loss_scale_window": 1000,\n "hysteresis": 2,\n "min_loss_scale": 1\n },\n\n # misc. training settings\n "train-iters": 320000,\n "lr-decay-iters": 320000,\n "distributed-backend": "nccl",\n "lr-decay-style": "cosine",\n "warmup": 0.01,\n "save-interval": 10000,\n "eval-interval": 1000,\n "eval-iters": 10,\n\n # logging\n "log-interval": 100,\n "steps_per_print": 10,\n "keep-last-n-checkpoints": 4,\n "wall_clock_breakdown": true,\n}\n', 'local_setup_small.yml': '# Suggested data paths when using GPT-NeoX locally\n{\n # "train-data-paths": ["/mnt/4TBNVME/data/the_pile_preprocessed/train/the_pile_text_document"],\n "data-path": "data/enron/enron_text_document",\n # "data-path": "/mnt/4TBNVME/data/the_pile_preprocessed/train/the_pile_text_document",\n # or for weighted datasets: \n # "train-data-paths": ["data/enron/enron_text_document", "data/enron/enron_text_document"],\n # "test-data-paths": ["data/enron/enron_text_document", "data/enron/enron_text_document"],\n # "valid-data-paths": ["data/enron/enron_text_document", "data/enron/enron_text_document"],\n # "train-data-weights": [1., 2.],\n # "test-data-weights": [2., 1.],\n # "valid-data-weights": [0.5, 0.4],\n\n # If weight_by_num_documents is True, Builds dataset weights from a multinomial distribution over groups of data according to the number of documents in each group. \n # WARNING: setting this to True will override any user provided weights\n # "weight_by_num_documents": false,\n # "weighted_sampler_alpha": 0.3,\n\n "vocab-file": "/mnt/4TBNVME/data/gpt2-vocab.json",\n "merge-file": "/mnt/4TBNVME/data/gpt2-merges.txt",\n\n "save": "checkpoints_small",\n "load": "checkpoints_small",\n "checkpoint_validation_with_forward_pass": False,\n \n "tensorboard-dir": "tensorboard/run4",\n "log-dir": "logs4",\n "use_wandb": False,\n "wandb_host": "https://api.wandb.ai",\n "wandb_project": "neox"\n}\n'}updated data_impl ....................... mmap........................updated data_path ....................... data/enron/enron_text_documentupdated dynamic_loss_scale .............. True........................updated eval_iters ...................... 10..........................updated fp16 ............................ {'enabled': True, 'loss_scale': 0, 'loss_scale_window': 1000, 'hysteresis': 2, 'min_loss_scale': 1}updated gas ............................. 1...........................updated global_num_gpus ................. 4...........................updated gradient_clipping ............... 1.0.........................updated hidden_dropout .................. 0.0.........................updated hidden_size ..................... 768.........................updated is_pipe_parallel ................ True........................updated keep_last_n_checkpoints ......... 4...........................updated load ............................ checkpoints_small...........updated log_dir ......................... logs4.......................updated log_interval .................... 100.........................updated lr .............................. 0.0006......................updated lr_decay_iters .................. 320000......................updated lr_decay_style .................. cosine......................updated max_position_embeddings ......... 2048........................updated merge_file ...................... /mnt/4TBNVME/data/gpt2-merges.txtupdated no_weight_tying ................. True........................updated num_attention_heads ............. 12..........................updated num_layers ...................... 12..........................updated optimizer ....................... {'type': 'Adam', 'params': {'lr': 0.0006, 'betas': [0.9, 0.999], 'eps': 1e-08}}updated optimizer_type .................. Adam........................updated partition_activations ........... True........................updated pipe_parallel_size .............. 1...........................updated pos_emb ......................... rotary......................updated precision ....................... fp16........................updated save ............................ checkpoints_small...........updated save_interval ................... 10000.......................updated seq_length ...................... 2048........................updated sparsity_config ................. {}..........................updated split ........................... 949,50,1....................updated synchronize_each_layer .......... True........................updated tensorboard_dir ................. tensorboard/run4............updated train_batch_size ................ 16..........................updated train_iters ..................... 320000......................updated train_micro_batch_size_per_gpu .. 4...........................updated use_wandb ....................... False.......................updated user_script ..................... train.py....................updated vocab_file ...................... /mnt/4TBNVME/data/gpt2-vocab.jsonupdated wall_clock_breakdown ............ True........................updated wandb_group ..................... YRCNBdcn62NVP5XPQgvHzC_3ml8m47dupdated weight_decay .................... 0.0.........................updated zero_allgather_bucket_size ...... 500000000...................updated zero_contiguous_gradients ....... True........................updated zero_optimization ............... {'stage': 2, 'allgather_partitions': True, 'allgather_bucket_size': 500000000, 'overlap_comm': True, 'reduce_scatter': True, 'reduce_bucket_size': 500000000, 'contiguous_gradients': True, 'cpu_offload': True}updated zero_reduce_bucket_size ......... 500000000...................updated zero_reduce_scatter ............. True........................updated zero_stage ...................... 2...........................updated activation ...................... gelu........................default adlr_autoresume ................. False.......................default adlr_autoresume_interval ........ 1000........................default amp ............................. None........................default apply_query_key_layer_scaling ... False.......................default attention_softmax_in_fp32 ....... False.......................default bias_dropout_fusion ............. False.......................default bias_gelu_fusion ................ False.......................default char_level_ppl .................. False.......................default checkpoint_in_cpu ............... False.......................default checkpoint_num_layers ........... 1...........................default checkpoint_validation_with_forward_pass False................default contiguous_checkpointing ........ False.......................default deepscale ....................... False.......................default deepscale_config ................ None........................default deepspeed ....................... True........................default deepspeed_activation_checkpointing True......................default deepspeed_mpi ................... False.......................default detect_nvlink_pairs ............. False.......................default distributed_backend ............. nccl........................default do_test ......................... None........................default do_train ........................ None........................default do_valid ........................ None........................default dump_state ...................... False.......................default eod_mask_loss ................... False.......................default eval_interval ................... 1000........................default eval_results_prefix ............. ............................default eval_tasks ...................... None........................default exclude ......................... None........................default exit_interval ................... None........................default finetune ........................ False.......................default flops_profiler .................. None........................default fp16_lm_cross_entropy ........... False.......................default fp32_allreduce .................. False.......................default git_hash ........................ f3a982d.....................default gmlp_attn_dim ................... 64..........................default gpt_j_residual .................. False.......................default gradient_accumulation_steps ..... 1...........................default gradient_noise_scale_cpu_offload False.......................default gradient_noise_scale_n_batches .. 5...........................default gradient_predivide_factor ....... 1.0.........................default hostfile ........................ None........................default hysteresis ...................... 2...........................default include ......................... None........................default init_method ..................... normal......................default init_method_std ................. 0.02........................default iteration ....................... None........................default launcher ........................ pdsh........................default layernorm_epsilon ............... 1e-05.......................default lazy_mpu_init ................... False.......................default local_rank ...................... None........................default log_grad_norm ................... False.......................default log_gradient_noise_scale ........ False.......................default log_optimizer_states ............ False.......................default log_param_norm .................. False.......................default loss_scale ...................... None........................default loss_scale_window ............... 1000.0......................default make_vocab_size_divisible_by .... 128.........................default master_addr ..................... None........................default master_port ..................... 29500.......................default maximum_tokens .................. 64..........................default min_lr .......................... 0.0.........................default min_scale ....................... 1.0.........................default mmap_warmup ..................... False.......................default model_parallel_size ............. 1...........................default no_load_optim ................... False.......................default no_load_rng ..................... False.......................default no_save_optim ................... False.......................default no_save_rng ..................... False.......................default norm ............................ layernorm...................default num_gpus ........................ None........................default num_nodes ....................... -1..........................default num_samples ..................... 0...........................default num_unique_layers ............... None........................default num_workers ..................... 2...........................default onnx_safe ....................... False.......................default output_layer_init_method ........ scaled_normal...............default output_layer_parallelism ........ row.........................default override_lr_scheduler ........... False.......................default padded_vocab_size ............... None........................default param_sharing_style ............. grouped.....................default pipe_partition_method ........... type:transformer|mlp........default prescale_gradients .............. False.......................default profile_backward ................ False.......................default rank ............................ None........................default recompute ....................... False.......................default rms_norm_epsilon ................ 1e-08.......................default rotary_emb_base ................. 10000.......................default rotary_pct ...................... 1.0.........................default rpe_max_distance ................ 128.........................default rpe_num_buckets ................. 32..........................default sample_input_file ............... None........................default sample_output_file .............. None........................default scaled_masked_softmax_fusion .... False.......................default scaled_upper_triang_masked_softmax_fusion False..............default scalenorm_epsilon ............... 1e-08.......................default scheduler ....................... None........................default seed ............................ 1234........................default short_seq_prob .................. 0.1.........................default soft_prompt_tuning .............. None........................default sparse_gradients ................ False.......................default steps_per_print ................. 10..........................default temperature ..................... 0.0.........................default test_data_paths ................. None........................default test_data_weights ............... None........................default text_gen_type ................... None........................default tokenizer_type .................. GPT2BPETokenizer............default top_k ........................... 0...........................default top_p ........................... 0.0.........................default train_data_paths ................ None........................default train_data_weights .............. None........................default use_bnb_optimizer ............... False.......................default use_checkpoint_lr_scheduler ..... False.......................default use_cpu_initialization .......... False.......................default valid_data_paths ................ None........................default valid_data_weights .............. None........................default wandb_host ...................... https://api.wandb.ai........default wandb_project ................... neox........................default wandb_team ...................... None........................default warmup .......................... 0.01........................default weight_by_num_documents ......... False.......................default weighted_sampler_alpha .......... 0.3.........................default world_size ...................... None........................default zero_allow_untested_optimizer ... False.......................default ---------------- end of arguments ---------------- [2021-12-10 19:56:34,044] [WARNING] [runner.py:126:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. [2021-12-10 19:56:34,044] [INFO] [runner.py:366:main] cmd = /root/venv/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgM119 --master_addr=127.0.0.1 --master_port=29500 train.py --deepspeed_config {"train_batch_size": 16, "train_micro_batch_size_per_gpu": 4, "optimizer": {"type": "Adam", "params": {"lr": 0.0006, "betas": [0.9, 0.999], "eps": 1e-08}}, "fp16": {"enabled": true, "loss_scale": 0, "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1}, "gradient_clipping": 1.0, "zero_optimization": {"stage": 2, "allgather_partitions": true, "allgather_bucket_size": 500000000, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 500000000, "contiguous_gradients": true, "cpu_offload": true}, "wall_clock_breakdown": true} --megatron_config {"train_batch_size": 16, "train_micro_batch_size_per_gpu": 4, "optimizer": {"type": "Adam", "params": {"lr": 0.0006, "betas": [0.9, 0.999], "eps": 1e-08}}, "fp16": {"enabled": true, "loss_scale": 0, "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1}, "gradient_clipping": 1.0, "zero_optimization": {"stage": 2, "allgather_partitions": true, "allgather_bucket_size": 500000000, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 500000000, "contiguous_gradients": true, "cpu_offload": true}, "wall_clock_breakdown": true, "precision": "fp16", "num_layers": 12, "hidden_size": 768, "num_attention_heads": 12, "seq_length": 2048, "max_position_embeddings": 2048, "pos_emb": "rotary", "no_weight_tying": true, "attention_config": ["global", "global", "global", "global", "global", "global", "global", "global", "global", "global", "global", "global"], "sparsity_config": {}, "lr_decay_style": "cosine", "lr_decay_iters": 320000, "optimizer_type": "Adam", "zero_stage": 2, "zero_reduce_scatter": true, "zero_contiguous_gradients": true, "zero_reduce_bucket_size": 500000000, "zero_allgather_bucket_size": 500000000, "lr": 0.0006, "data_path": "data/enron/enron_text_document", "data_impl": "mmap", "save": "checkpoints_small", "config_files": {"small.yml": "# GPT-2 pretraining setup\n{\n # parallelism settings ( you will want to change these based on your cluster setup, ideally scheduling pipeline stages\n # across the node boundaries )\n \"pipe-parallel-size\": 1,\n \"model-parallel-size\": 1,\n\n # model settings\n \"num-layers\": 12,\n \"hidden-size\": 768,\n \"num-attention-heads\": 12,\n \"seq-length\": 2048,\n \"max-position-embeddings\": 2048,\n \"norm\": \"layernorm\",\n \"pos-emb\": \"rotary\",\n \"no-weight-tying\": true,\n\n # these should provide some speedup but takes a while to build, set to true if desired\n \"scaled-upper-triang-masked-softmax-fusion\": false,\n \"bias-gelu-fusion\": false,\n\n\n # optimizer settings\n \"optimizer\": {\n \"type\": \"Adam\",\n \"params\": {\n \"lr\": 0.0006,\n \"betas\": [0.9, 0.999],\n \"eps\": 1.0e-8,\n }\n },\n \"zero_optimization\": {\n \"stage\": 2,\n \"allgather_partitions\": True,\n \"allgather_bucket_size\": 500000000,\n \"overlap_comm\": True,\n \"reduce_scatter\": True,\n \"reduce_bucket_size\": 500000000,\n \"contiguous_gradients\": True,\n \"cpu_offload\": True\n },\n\n # batch / data settings\n \"train_micro_batch_size_per_gpu\": 4,\n \"data-impl\": \"mmap\",\n \"split\": \"949,50,1\",\n\n # activation checkpointing\n \"checkpoint-activations\": true,\n \"checkpoint-num-layers\": 1,\n \"partition-activations\": true,\n \"synchronize-each-layer\": true,\n\n # regularization\n \"gradient_clipping\": 1.0,\n \"weight-decay\": 0.0,\n \"hidden-dropout\": 0.0,\n \"attention-dropout\": 0.0,\n\n # precision settings\n \"fp16\": { \n \"enabled\": true,\n \"loss_scale\": 0,\n \"loss_scale_window\": 1000,\n \"hysteresis\": 2,\n \"min_loss_scale\": 1\n },\n\n # misc. training settings\n \"train-iters\": 320000,\n \"lr-decay-iters\": 320000,\n \"distributed-backend\": \"nccl\",\n \"lr-decay-style\": \"cosine\",\n \"warmup\": 0.01,\n \"save-interval\": 10000,\n \"eval-interval\": 1000,\n \"eval-iters\": 10,\n\n # logging\n \"log-interval\": 100,\n \"steps_per_print\": 10,\n \"keep-last-n-checkpoints\": 4,\n \"wall_clock_breakdown\": true,\n}\n", "local_setup_small.yml": "# Suggested data paths when using GPT-NeoX locally\n{\n # \"train-data-paths\": [\"/mnt/4TBNVME/data/the_pile_preprocessed/train/the_pile_text_document\"],\n \"data-path\": \"data/enron/enron_text_document\",\n # \"data-path\": \"/mnt/4TBNVME/data/the_pile_preprocessed/train/the_pile_text_document\",\n # or for weighted datasets: \n # \"train-data-paths\": [\"data/enron/enron_text_document\", \"data/enron/enron_text_document\"],\n # \"test-data-paths\": [\"data/enron/enron_text_document\", \"data/enron/enron_text_document\"],\n # \"valid-data-paths\": [\"data/enron/enron_text_document\", \"data/enron/enron_text_document\"],\n # \"train-data-weights\": [1., 2.],\n # \"test-data-weights\": [2., 1.],\n # \"valid-data-weights\": [0.5, 0.4],\n\n # If weight_by_num_documents is True, Builds dataset weights from a multinomial distribution over groups of data according to the number of documents in each group. \n # WARNING: setting this to True will override any user provided weights\n # \"weight_by_num_documents\": false,\n # \"weighted_sampler_alpha\": 0.3,\n\n \"vocab-file\": \"/mnt/4TBNVME/data/gpt2-vocab.json\",\n \"merge-file\": \"/mnt/4TBNVME/data/gpt2-merges.txt\",\n\n \"save\": \"checkpoints_small\",\n \"load\": \"checkpoints_small\",\n \"checkpoint_validation_with_forward_pass\": False,\n \n \"tensorboard-dir\": \"tensorboard/run4\",\n \"log-dir\": \"logs4\",\n \"use_wandb\": False,\n \"wandb_host\": \"https://api.wandb.ai\",\n \"wandb_project\": \"neox\"\n}\n"}, "load": "checkpoints_small", "save_interval": 10000, "batch_size": 4, "train_iters": 320000, "eval_iters": 10, "keep_last_n_checkpoints": 4, "split": "949,50,1", "vocab_file": "/mnt/4TBNVME/data/gpt2-vocab.json", "merge_file": "/mnt/4TBNVME/data/gpt2-merges.txt", "attention_dropout": 0.0, "hidden_dropout": 0.0, "weight_decay": 0.0, "checkpoint_activations": true, "synchronize_each_layer": true, "partition_activations": true, "gas": 1, "clip_grad": 1.0, "dynamic_loss_scale": true, "pipe_parallel_size": 1, "is_pipe_parallel": true, "use_wandb": false, "wandb_group": "YRCNBdcn62NVP5XPQgvHzC_3ml8m47d", "log_dir": "logs4", "tensorboard_dir": "tensorboard/run4", "log_interval": 100, "user_script": "train.py", "global_num_gpus": 4} [2021-12-10 19:56:34,871] [INFO] [launch.py:75:main] 0 NV_LIBNCCL_DEV_PACKAGE libnccl-dev=2.8.4-1+cuda11.1 [2021-12-10 19:56:34,871] [INFO] [launch.py:75:main] 0 NCCL_VERSION 2.8.4-1 [2021-12-10 19:56:34,871] [INFO] [launch.py:75:main] 0 NV_LIBNCCL_PACKAGE_VERSION 2.8.4-1 [2021-12-10 19:56:34,871] [INFO] [launch.py:75:main] 0 NV_LIBNCCL_PACKAGE libnccl2=2.8.4-1+cuda11.1 [2021-12-10 19:56:34,871] [INFO] [launch.py:75:main] 0 NV_LIBNCCL_DEV_PACKAGE_NAME libnccl-dev [2021-12-10 19:56:34,871] [INFO] [launch.py:75:main] 0 NV_LIBNCCL_PACKAGE_NAME libnccl2 [2021-12-10 19:56:34,871] [INFO] [launch.py:75:main] 0 NV_LIBNCCL_DEV_PACKAGE_VERSION 2.8.4-1 [2021-12-10 19:56:34,871] [INFO] [launch.py:82:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3]} [2021-12-10 19:56:34,871] [INFO] [launch.py:88:main] nnodes=1, num_local_procs=4, node_rank=0 [2021-12-10 19:56:34,871] [INFO] [launch.py:103:main] global_rank_mapping=defaultdict(, {'localhost': [0, 1, 2, 3]})
[2021-12-10 19:56:34,872] [INFO] [launch.py:104:main] dist_world_size=4
[2021-12-10 19:56:34,872] [INFO] [launch.py:112:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
NeoXArgs.configure_distributed_args() using world size: 4 and model-parallel size: 1
> building GPT2BPETokenizer tokenizer ...
> padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
> setting tensorboard ...
> initializing torch distributed ...
[2021-12-10 19:56:37,852] [INFO] [distributed.py:46:init_distributed] Initializing torch distributed with backend: nccl
[2021-12-10 19:56:37,873] [INFO] [distributed.py:46:init_distributed] Initializing torch distributed with backend: nccl
[2021-12-10 19:56:37,905] [INFO] [distributed.py:46:init_distributed] Initializing torch distributed with backend: nccl
[2021-12-10 19:56:38,017] [INFO] [distributed.py:46:init_distributed] Initializing torch distributed with backend: nccl
> initializing model parallel with size 1
MPU DP: [0, 1, 2, 3]
MPU PP: [0]
MPU PP: [1]
MPU PP: [2]
MPU PP: [3]
MPU MP: [0]
MPU MP: [1]
MPU MP: [2]
MPU MP: [3]
> setting random seeds to 1234 ...
[2021-12-10 19:56:38,112] [INFO] [checkpointing.py:223:model_parallel_cuda_manual_seed] > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234
make: Entering directory '/mnt/4TBNVME/gpt-neox/megatron/data'
make: Nothing to be done for 'default'.
make: Leaving directory '/mnt/4TBNVME/gpt-neox/megatron/data'
building GPT2 model ...
SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None
Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3}
[2021-12-10 19:56:41,471] [INFO] [module.py:363:_partition_layers] Partitioning pipeline stages with method type:transformer|mlp
stage=0 layers=17
0: EmbeddingPipe
1: _pre_transformer_block
2: ParallelTransformerLayerPipe
3: ParallelTransformerLayerPipe
4: ParallelTransformerLayerPipe
5: ParallelTransformerLayerPipe
6: ParallelTransformerLayerPipe
7: ParallelTransformerLayerPipe
8: ParallelTransformerLayerPipe
9: ParallelTransformerLayerPipe
10: ParallelTransformerLayerPipe
11: ParallelTransformerLayerPipe
12: ParallelTransformerLayerPipe
13: ParallelTransformerLayerPipe
14: _post_transformer_block
15: NormPipe
16: ParallelLinearPipe
loss: partial
[2021-12-10 19:56:41,497] [WARNING] [config.py:77:_sanity_check] DeepSpeedConfig: cpu_offload is deprecated. Please use offload_optimizer.
Configuring Optimizer type: Adam with params: {'lr': 0.0006, 'betas': [0.9, 0.999], 'eps': 1e-08}
> learning rate decay style: cosine
DeepSpeed is enabled.
[2021-12-10 19:56:41,502] [WARNING] [config.py:77:_sanity_check] DeepSpeedConfig: cpu_offload is deprecated. Please use offload_optimizer.
[2021-12-10 19:56:41,502] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.15+eb7f5cf, git-hash=eb7f5cf, git-branch=main
[2021-12-10 19:56:41,502] [WARNING] [config.py:77:_sanity_check] DeepSpeedConfig: cpu_offload is deprecated. Please use offload_optimizer.
[2021-12-10 19:56:41,506] [WARNING] [config.py:77:_sanity_check] DeepSpeedConfig: cpu_offload is deprecated. Please use offload_optimizer.
Using /root/.cache/torch_extensions as PyTorch extensions root...
[2021-12-10 19:56:42,718] [INFO] [engine.py:654:_configure_optimizer] Removing param_group that has no 'params' in the client Optimizer
[2021-12-10 19:56:42,718] [INFO] [engine.py:659:_configure_optimizer] Using client Optimizer as basic optimizer
[2021-12-10 19:56:42,718] [INFO] [engine.py:668:_configure_optimizer] DeepSpeed Basic Optimizer = FusedAdam
Checking ZeRO support for optimizer=FusedAdam type=
[2021-12-10 19:56:42,718] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
[2021-12-10 19:56:42,718] [INFO] [stage2.py:102:__init__] Reduce bucket size 500000000
[2021-12-10 19:56:42,718] [INFO] [stage2.py:103:__init__] Allgather bucket size 500000000
[2021-12-10 19:56:42,718] [INFO] [stage2.py:104:__init__] CPU Offload: True
Using /root/.cache/torch_extensions as PyTorch extensions root...
Using /root/.cache/torch_extensions as PyTorch extensions root...
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module utils...
Time to load utils op: 0.7915315628051758 seconds
Loading extension module utils...
Loading extension module utils...
Loading extension module utils...
Time to load utils op: 0.8036620616912842 seconds
Time to load utils op: 0.8034920692443848 seconds
Time to load utils op: 0.8034873008728027 seconds
Traceback (most recent call last):
File "train.py", line 27, in
pretrain(neox_args=neox_args)
File "/mnt/4TBNVME/gpt-neox/megatron/training.py", line 82, in pretrain
model, optimizer, lr_scheduler = setup_model_and_optimizer(
File "/mnt/4TBNVME/gpt-neox/megatron/training.py", line 399, in setup_model_and_optimizer
model, optimizer, _, lr_scheduler = deepspeed.initialize(
File "/root/venv/lib/python3.8/site-packages/deepspeed/__init__.py", line 128, in initialize
engine = PipelineEngine(args=args,
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 60, in __init__
super().__init__(*super_args, **super_kwargs)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 192, in __init__
self._configure_optimizer(optimizer, model_parameters)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 681, in _configure_optimizer
self.optimizer = self._configure_zero_optimizer(basic_optimizer)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 824, in _configure_zero_optimizer
optimizer = FP16_DeepSpeedZeroOptimizer(
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 373, in __init__
self.initialize_optimizer_states()
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 399, in initialize_optimizer_states
self.optimizer.step()
File "/root/venv/lib/python3.8/site-packages/torch/optim/optimizer.py", line 89, in wrapper
return func(*args, **kwargs)
File "/root/venv/lib/python3.8/site-packages/apex/optimizers/fused_adam.py", line 160, in step
multi_tensor_applier(self.multi_tensor_adam,
File "/root/venv/lib/python3.8/site-packages/apex/multi_tensor_apply/multi_tensor_apply.py", line 27, in __call__
return op(self.chunk_size,
RuntimeError: expected input to be on cuda
Traceback (most recent call last):
File "train.py", line 27, in
pretrain(neox_args=neox_args)
File "/mnt/4TBNVME/gpt-neox/megatron/training.py", line 82, in pretrain
model, optimizer, lr_scheduler = setup_model_and_optimizer(
File "/mnt/4TBNVME/gpt-neox/megatron/training.py", line 399, in setup_model_and_optimizer
model, optimizer, _, lr_scheduler = deepspeed.initialize(
File "/root/venv/lib/python3.8/site-packages/deepspeed/__init__.py", line 128, in initialize
engine = PipelineEngine(args=args,
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 60, in __init__
super().__init__(*super_args, **super_kwargs)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 192, in __init__
self._configure_optimizer(optimizer, model_parameters)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 681, in _configure_optimizer
self.optimizer = self._configure_zero_optimizer(basic_optimizer)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 824, in _configure_zero_optimizer
optimizer = FP16_DeepSpeedZeroOptimizer(
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 373, in __init__
self.initialize_optimizer_states()
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 399, in initialize_optimizer_states
self.optimizer.step()
File "/root/venv/lib/python3.8/site-packages/torch/optim/optimizer.py", line 89, in wrapper
return func(*args, **kwargs)
File "/root/venv/lib/python3.8/site-packages/apex/optimizers/fused_adam.py", line 160, in step
multi_tensor_applier(self.multi_tensor_adam,
File "/root/venv/lib/python3.8/site-packages/apex/multi_tensor_apply/multi_tensor_apply.py", line 27, in __call__
return op(self.chunk_size,
RuntimeError: expected input to be on cuda
Traceback (most recent call last):
File "train.py", line 27, in
pretrain(neox_args=neox_args)
File "/mnt/4TBNVME/gpt-neox/megatron/training.py", line 82, in pretrain
model, optimizer, lr_scheduler = setup_model_and_optimizer(
File "/mnt/4TBNVME/gpt-neox/megatron/training.py", line 399, in setup_model_and_optimizer
model, optimizer, _, lr_scheduler = deepspeed.initialize(
File "/root/venv/lib/python3.8/site-packages/deepspeed/__init__.py", line 128, in initialize
engine = PipelineEngine(args=args,
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 60, in __init__
super().__init__(*super_args, **super_kwargs)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 192, in __init__
self._configure_optimizer(optimizer, model_parameters)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 681, in _configure_optimizer
self.optimizer = self._configure_zero_optimizer(basic_optimizer)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 824, in _configure_zero_optimizer
optimizer = FP16_DeepSpeedZeroOptimizer(
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 373, in __init__
self.initialize_optimizer_states()
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 399, in initialize_optimizer_states
self.optimizer.step()
File "/root/venv/lib/python3.8/site-packages/torch/optim/optimizer.py", line 89, in wrapper
return func(*args, **kwargs)
File "/root/venv/lib/python3.8/site-packages/apex/optimizers/fused_adam.py", line 160, in step
multi_tensor_applier(self.multi_tensor_adam,
File "/root/venv/lib/python3.8/site-packages/apex/multi_tensor_apply/multi_tensor_apply.py", line 27, in __call__
return op(self.chunk_size,
RuntimeError: expected input to be on cuda
Traceback (most recent call last):
File "train.py", line 27, in
pretrain(neox_args=neox_args)
File "/mnt/4TBNVME/gpt-neox/megatron/training.py", line 82, in pretrain
model, optimizer, lr_scheduler = setup_model_and_optimizer(
File "/mnt/4TBNVME/gpt-neox/megatron/training.py", line 399, in setup_model_and_optimizer
model, optimizer, _, lr_scheduler = deepspeed.initialize(
File "/root/venv/lib/python3.8/site-packages/deepspeed/__init__.py", line 128, in initialize
engine = PipelineEngine(args=args,
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 60, in __init__
super().__init__(*super_args, **super_kwargs)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 192, in __init__
self._configure_optimizer(optimizer, model_parameters)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 681, in _configure_optimizer
self.optimizer = self._configure_zero_optimizer(basic_optimizer)
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 824, in _configure_zero_optimizer
optimizer = FP16_DeepSpeedZeroOptimizer(
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 373, in __init__
self.initialize_optimizer_states()
File "/root/venv/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 399, in initialize_optimizer_states
self.optimizer.step()
File "/root/venv/lib/python3.8/site-packages/torch/optim/optimizer.py", line 89, in wrapper
return func(*args, **kwargs)
File "/root/venv/lib/python3.8/site-packages/apex/optimizers/fused_adam.py", line 160, in step
multi_tensor_applier(self.multi_tensor_adam,
File "/root/venv/lib/python3.8/site-packages/apex/multi_tensor_apply/multi_tensor_apply.py", line 27, in __call__
return op(self.chunk_size,
RuntimeError: expected input to be on cuda
Killing subprocess 3126
Killing subprocess 3127
Killing subprocess 3128
Killing subprocess 3130
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/venv/lib/python3.8/site-packages/deepspeed/launcher/launch.py", line 179, in
main()
File "/root/venv/lib/python3.8/site-packages/deepspeed/launcher/launch.py", line 169, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/root/venv/lib/python3.8/site-packages/deepspeed/launcher/launch.py", line 147, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/root/venv/bin/python', '-u', 'train.py', '--local_rank=3', '--deepspeed_config', '{"train_batch_size": 16, "train_micro_batch_size_per_gpu": 4, "optimizer": {"type": "Adam", "params": {"lr": 0.0006, "betas": [0.9, 0.999], "eps": 1e-08}}, "fp16": {"enabled": true, "loss_scale": 0, "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1}, "gradient_clipping": 1.0, "zero_optimization": {"stage": 2, "allgather_partitions": true, "allgather_bucket_size": 500000000, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 500000000, "contiguous_gradients": true, "cpu_offload": true}, "wall_clock_breakdown": true}', '--megatron_config', '{"train_batch_size": 16, "train_micro_batch_size_per_gpu": 4, "optimizer": {"type": "Adam", "params": {"lr": 0.0006, "betas": [0.9, 0.999], "eps": 1e-08}}, "fp16": {"enabled": true, "loss_scale": 0, "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1}, "gradient_clipping": 1.0, "zero_optimization": {"stage": 2, "allgather_partitions": true, "allgather_bucket_size": 500000000, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 500000000, "contiguous_gradients": true, "cpu_offload": true}, "wall_clock_breakdown": true, "precision": "fp16", "num_layers": 12, "hidden_size": 768, "num_attention_heads": 12, "seq_length": 2048, "max_position_embeddings": 2048, "pos_emb": "rotary", "no_weight_tying": true, "attention_config": ["global", "global", "global", "global", "global", "global", "global", "global", "global", "global", "global", "global"], "sparsity_config": {}, "lr_decay_style": "cosine", "lr_decay_iters": 320000, "optimizer_type": "Adam", "zero_stage": 2, "zero_reduce_scatter": true, "zero_contiguous_gradients": true, "zero_reduce_bucket_size": 500000000, "zero_allgather_bucket_size": 500000000, "lr": 0.0006, "data_path": "data/enron/enron_text_document", "data_impl": "mmap", "save": "checkpoints_small", "config_files": {"small.yml": "# GPT-2 pretraining setup\\n{\\n # parallelism settings ( you will want to change these based on your cluster setup, ideally scheduling pipeline stages\\n # across the node boundaries )\\n \\"pipe-parallel-size\\": 1,\\n \\"model-parallel-size\\": 1,\\n\\n # model settings\\n \\"num-layers\\": 12,\\n \\"hidden-size\\": 768,\\n \\"num-attention-heads\\": 12,\\n \\"seq-length\\": 2048,\\n \\"max-position-embeddings\\": 2048,\\n \\"norm\\": \\"layernorm\\",\\n \\"pos-emb\\": \\"rotary\\",\\n \\"no-weight-tying\\": true,\\n\\n # these should provide some speedup but takes a while to build, set to true if desired\\n \\"scaled-upper-triang-masked-softmax-fusion\\": false,\\n \\"bias-gelu-fusion\\": false,\\n\\n\\n # optimizer settings\\n \\"optimizer\\": {\\n \\"type\\": \\"Adam\\",\\n \\"params\\": {\\n \\"lr\\": 0.0006,\\n \\"betas\\": [0.9, 0.999],\\n \\"eps\\": 1.0e-8,\\n }\\n },\\n \\"zero_optimization\\": {\\n \\"stage\\": 2,\\n \\"allgather_partitions\\": True,\\n \\"allgather_bucket_size\\": 500000000,\\n \\"overlap_comm\\": True,\\n \\"reduce_scatter\\": True,\\n \\"reduce_bucket_size\\": 500000000,\\n \\"contiguous_gradients\\": True,\\n \\"cpu_offload\\": True\\n },\\n\\n # batch / data settings\\n \\"train_micro_batch_size_per_gpu\\": 4,\\n \\"data-impl\\": \\"mmap\\",\\n \\"split\\": \\"949,50,1\\",\\n\\n # activation checkpointing\\n \\"checkpoint-activations\\": true,\\n \\"checkpoint-num-layers\\": 1,\\n \\"partition-activations\\": true,\\n \\"synchronize-each-layer\\": true,\\n\\n # regularization\\n \\"gradient_clipping\\": 1.0,\\n \\"weight-decay\\": 0.0,\\n \\"hidden-dropout\\": 0.0,\\n \\"attention-dropout\\": 0.0,\\n\\n # precision settings\\n \\"fp16\\": { \\n \\"enabled\\": true,\\n \\"loss_scale\\": 0,\\n \\"loss_scale_window\\": 1000,\\n \\"hysteresis\\": 2,\\n \\"min_loss_scale\\": 1\\n },\\n\\n # misc. training settings\\n \\"train-iters\\": 320000,\\n \\"lr-decay-iters\\": 320000,\\n \\"distributed-backend\\": \\"nccl\\",\\n \\"lr-decay-style\\": \\"cosine\\",\\n \\"warmup\\": 0.01,\\n \\"save-interval\\": 10000,\\n \\"eval-interval\\": 1000,\\n \\"eval-iters\\": 10,\\n\\n # logging\\n \\"log-interval\\": 100,\\n \\"steps_per_print\\": 10,\\n \\"keep-last-n-checkpoints\\": 4,\\n \\"wall_clock_breakdown\\": true,\\n}\\n", "local_setup_small.yml": "# Suggested data paths when using GPT-NeoX locally\\n{\\n # \\"train-data-paths\\": [\\"/mnt/4TBNVME/data/the_pile_preprocessed/train/the_pile_text_document\\"],\\n \\"data-path\\": \\"data/enron/enron_text_document\\",\\n # \\"data-path\\": \\"/mnt/4TBNVME/data/the_pile_preprocessed/train/the_pile_text_document\\",\\n # or for weighted datasets: \\n # \\"train-data-paths\\": [\\"data/enron/enron_text_document\\", \\"data/enron/enron_text_document\\"],\\n # \\"test-data-paths\\": [\\"data/enron/enron_text_document\\", \\"data/enron/enron_text_document\\"],\\n # \\"valid-data-paths\\": [\\"data/enron/enron_text_document\\", \\"data/enron/enron_text_document\\"],\\n # \\"train-data-weights\\": [1., 2.],\\n # \\"test-data-weights\\": [2., 1.],\\n # \\"valid-data-weights\\": [0.5, 0.4],\\n\\n # If weight_by_num_documents is True, Builds dataset weights from a multinomial distribution over groups of data according to the number of documents in each group. \\n # WARNING: setting this to True will override any user provided weights\\n # \\"weight_by_num_documents\\": false,\\n # \\"weighted_sampler_alpha\\": 0.3,\\n\\n \\"vocab-file\\": \\"/mnt/4TBNVME/data/gpt2-vocab.json\\",\\n \\"merge-file\\": \\"/mnt/4TBNVME/data/gpt2-merges.txt\\",\\n\\n \\"save\\": \\"checkpoints_small\\",\\n \\"load\\": \\"checkpoints_small\\",\\n \\"checkpoint_validation_with_forward_pass\\": False,\\n \\n \\"tensorboard-dir\\": \\"tensorboard/run4\\",\\n \\"log-dir\\": \\"logs4\\",\\n \\"use_wandb\\": False,\\n \\"wandb_host\\": \\"https://api.wandb.ai\\",\\n \\"wandb_project\\": \\"neox\\"\\n}\\n"}, "load": "checkpoints_small", "save_interval": 10000, "batch_size": 4, "train_iters": 320000, "eval_iters": 10, "keep_last_n_checkpoints": 4, "split": "949,50,1", "vocab_file": "/mnt/4TBNVME/data/gpt2-vocab.json", "merge_file": "/mnt/4TBNVME/data/gpt2-merges.txt", "attention_dropout": 0.0, "hidden_dropout": 0.0, "weight_decay": 0.0, "checkpoint_activations": true, "synchronize_each_layer": true, "partition_activations": true, "gas": 1, "clip_grad": 1.0, "dynamic_loss_scale": true, "pipe_parallel_size": 1, "is_pipe_parallel": true, "use_wandb": false, "wandb_group": "YRCNBdcn62NVP5XPQgvHzC_3ml8m47d", "log_dir": "logs4", "tensorboard_dir": "tensorboard/run4", "log_interval": 100, "user_script": "train.py", "global_num_gpus": 4}']' returned non-zero exit status 1.
To Reproduce
Starting on a host with NVIDIA driver 460.91.03, I ran a new docker container using image
nvidia/cuda:11.1.1-devel-ubuntu18.04. I created a new Python 3.8.0 virtual env, installed torch 1.8.2+cu111, installed the latest NVIDIA apex, then installed all the requirements in each file within the requirements directory of GPT-NeoX.I changed two lines in the
small.ymlconfig in the"zero_optimization"section:"stage": 2"cpu_offload": TrueThen I ran:
This caused the mentioned error.
Expected behavior
The optimizer should be offloaded to the CPU.
Proposed solution
No proposed solutions yet. Are there any known working
cpu_offloadconfigs? I can start there to try and debug this.Screenshots
None
Environment (please complete the following information):
small.yml(modified)local_setup_small.ymlAdditional context
None