[Bert/Pytorch] pretraining FileNotFoundError

[jzhang82119]

I think I finished the step 5 /workspace/bert/data/create_datasets_from_start.sh in quick start guide. It took a whole day or so.

Now I am trying to do bash scripts/run_pretraining.sh benchmark.

Below is the output.

Container nvidia build = 13419386 Warning! /workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/ directory missing. Training cannot start /workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/ Logs written to /workspace/bert/results/bert_lamb_pretraining.pyt_bert_pretraining_phase1_fp16_gbs65536.210227234326.log

• python3 -m torch.distributed.launch --nproc_per_node=8 /workspace/bert/run_pretraining.py --input_dir=/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/ --output_dir=/workspace/bert/results/checkpoints --config_file=bert_config.json --bert_model=bert-large-uncased --train_batch_size=8192 --max_seq_length=128 --max_predictions_per_seq=20 --max_steps=7038 --warmup_proportion=0.2843 --num_steps_per_checkpoint=200 --learning_rate=6e-3 --seed=12439 --fp16 --gradient_accumulation_steps=128 --allreduce_post_accumulation --allreduce_post_accumulation_fp16 --do_train --json-summary /workspace/bert/results/dllogger.json device: cuda:3 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:7 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:5 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:2 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:6 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:1 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:4 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:0 n_gpu: 1, distributed training: True, 16-bits training: True DLL 2021-02-27 23:43:30.868704 - PARAMETER Config : ["Namespace(allreduce_post_accumulation=True, allreduce_post_accumulation_fp16=True, amp=False, bert_model='bert-large-uncased', checkpoint_activations=False, config_file='bert_config.json', disable_progress_bar=False, do_train=True, fp16=True, gradient_accumulation_steps=128, init_checkpoint=None, init_loss_scale=1048576, input_dir='/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/', json_summary='/workspace/bert/results/dllogger.json', learning_rate=0.006, local_rank=0, log_freq=1.0, loss_scale=0.0, max_predictions_per_seq=20, max_seq_length=128, max_steps=7038.0, n_gpu=1, num_steps_per_checkpoint=200, num_train_epochs=3.0, output_dir='/workspace/bert/results/checkpoints', phase1_end_step=7038, phase2=False, resume_from_checkpoint=False, resume_step=-1, seed=12439, skip_checkpoint=False, steps_this_run=7038.0, train_batch_size=64, use_env=False, warmup_proportion=0.2843)"] Selected optimization level O2: FP16 training with FP32 batchnorm and FP32 master weights.

Defaults for this optimization level are: enabled : True opt_level : O2 cast_model_type : torch.float16 patch_torch_functions : False keep_batchnorm_fp32 : True master_weights : True loss_scale : dynamic Processing user overrides (additional kwargs that are not None)... After processing overrides, optimization options are: enabled : True opt_level : O2 cast_model_type : torch.float16 patch_torch_functions : False keep_batchnorm_fp32 : True master_weights : True loss_scale : dynamic Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 531, in main files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if FileNotFoundError: [Errno 2] No such file or directory: '/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/' DLL 2021-02-27 23:43:44.783208 - PARAMETER SEED : 12439 DLL 2021-02-27 23:43:44.783634 - PARAMETER train_start : True DLL 2021-02-27 23:43:44.783717 - PARAMETER batch_size_per_gpu : 64 DLL 2021-02-27 23:43:44.783760 - PARAMETER learning_rate : 0.006 Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 531, in main Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if FileNotFoundError: [Errno 2] No such file or directory: '/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/' args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 531, in main files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if FileNotFoundError: [Errno 2] No such file or directory: '/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/' Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 531, in main files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if FileNotFoundError: [Errno 2] No such file or directory: '/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/' Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 531, in main files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 531, in main args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 531, in main FileNotFoundError: [Errno 2] No such file or directory: '/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/' files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if FileNotFoundError: [Errno 2] No such file or directory: '/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/' FileNotFoundError: [Errno 2] No such file or directory: '/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/' Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 531, in main files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if FileNotFoundError: [Errno 2] No such file or directory: '/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/' Traceback (most recent call last): File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in main() File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main cmd=cmd) subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-u', '/workspace/bert/run_pretraining.py', '--local_rank=7', '--input_dir=/workspace/bert/data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/', '--output_dir=/workspace/bert/results/checkpoints', '--config_file=bert_config.json', '--bert_model=bert-large-uncased', '--train_batch_size=8192', '--max_seq_length=128', '--max_predictions_per_seq=20', '--max_steps=7038', '--warmup_proportion=0.2843', '--num_steps_per_checkpoint=200', '--learning_rate=6e-3', '--seed=12439', '--fp16', '--gradient_accumulation_steps=128', '--allreduce_post_accumulation', '--allreduce_post_accumulation_fp16', '--do_train', '--json-summary', '/workspace/bert/results/dllogger.json']' returned non-zero exit status 1.

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

finished pretraining /workspace/bert/data/hdf5_lower_case_1_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/ Logs written to /workspace/bert/results/bert_lamb_pretraining.pyt_bert_pretraining_phase2_fp16_gbs32768.210227234346.log

• python3 -m torch.distributed.launch --nproc_per_node=8 /workspace/bert/run_pretraining.py --input_dir=/workspace/bert/data/hdf5_lower_case_1_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/ --output_dir=/workspace/bert/results/checkpoints --config_file=bert_config.json --bert_model=bert-large-uncased --train_batch_size=4096 --max_seq_length=512 --max_predictions_per_seq=80 --max_steps=1563 --warmup_proportion=0.128 --num_steps_per_checkpoint=200 --learning_rate=4e-3 --seed=12439 --fp16 --gradient_accumulation_steps=512 --allreduce_post_accumulation --allreduce_post_accumulation_fp16 --do_train --phase2 --resume_from_checkpoint --phase1_end_step=7038 --json-summary /workspace/bert/results/dllogger.json device: cuda:2 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:1 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:4 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:5 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:7 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:3 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:6 n_gpu: 1, distributed training: True, 16-bits training: True device: cuda:0 n_gpu: 1, distributed training: True, 16-bits training: True DLL 2021-02-27 23:43:50.979698 - PARAMETER Config : ["Namespace(allreduce_post_accumulation=True, allreduce_post_accumulation_fp16=True, amp=False, bert_model='bert-large-uncased', checkpoint_activations=False, config_file='bert_config.json', disable_progress_bar=False, do_train=True, fp16=True, gradient_accumulation_steps=512, init_checkpoint=None, init_loss_scale=1048576, input_dir='/workspace/bert/data/hdf5_lower_case_1_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/', json_summary='/workspace/bert/results/dllogger.json', learning_rate=0.004, local_rank=0, log_freq=1.0, loss_scale=0.0, max_predictions_per_seq=80, max_seq_length=512, max_steps=1563.0, n_gpu=1, num_steps_per_checkpoint=200, num_train_epochs=3.0, output_dir='/workspace/bert/results/checkpoints', phase1_end_step=7038, phase2=True, resume_from_checkpoint=True, resume_step=-1, seed=12439, skip_checkpoint=False, steps_this_run=1563.0, train_batch_size=8, use_env=False, warmup_proportion=0.128)"] Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 506, in main model, optimizer, lr_scheduler, checkpoint, global_step, criterion = prepare_model_and_optimizer(args, device) File "/workspace/bert/run_pretraining.py", line 359, in prepare_model_and_optimizer args.resumestep = max([int(x.split('.pt')[0].split('')[1].strip()) for x in model_names]) ValueError: max() arg is an empty sequence Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 506, in main model, optimizer, lr_scheduler, checkpoint, global_step, criterion = prepare_model_and_optimizer(args, device) File "/workspace/bert/run_pretraining.py", line 359, in prepare_model_and_optimizer args.resumestep = max([int(x.split('.pt')[0].split('')[1].strip()) for x in model_names]) ValueError: max() arg is an empty sequence Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 506, in main model, optimizer, lr_scheduler, checkpoint, global_step, criterion = prepare_model_and_optimizer(args, device) File "/workspace/bert/run_pretraining.py", line 359, in prepare_model_and_optimizer args.resumestep = max([int(x.split('.pt')[0].split('')[1].strip()) for x in model_names]) ValueError: max() arg is an empty sequence Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 506, in main model, optimizer, lr_scheduler, checkpoint, global_step, criterion = prepare_model_and_optimizer(args, device) File "/workspace/bert/run_pretraining.py", line 359, in prepare_model_and_optimizer args.resumestep = max([int(x.split('.pt')[0].split('')[1].strip()) for x in model_names]) ValueError: max() arg is an empty sequence Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 506, in main model, optimizer, lr_scheduler, checkpoint, global_step, criterion = prepare_model_and_optimizer(args, device) File "/workspace/bert/run_pretraining.py", line 359, in prepare_model_and_optimizer args.resumestep = max([int(x.split('.pt')[0].split('')[1].strip()) for x in model_names]) ValueError: max() arg is an empty sequence Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 506, in main model, optimizer, lr_scheduler, checkpoint, global_step, criterion = prepare_model_and_optimizer(args, device) File "/workspace/bert/run_pretraining.py", line 359, in prepare_model_and_optimizer args.resumestep = max([int(x.split('.pt')[0].split('')[1].strip()) for x in model_names]) ValueError: max() arg is an empty sequence Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 506, in main model, optimizer, lr_scheduler, checkpoint, global_step, criterion = prepare_model_and_optimizer(args, device) File "/workspace/bert/run_pretraining.py", line 359, in prepare_model_and_optimizer args.resumestep = max([int(x.split('.pt')[0].split('')[1].strip()) for x in model_names]) ValueError: max() arg is an empty sequence Traceback (most recent call last): File "/workspace/bert/run_pretraining.py", line 678, in args, final_loss, train_time_raw, global_step = main() File "/workspace/bert/run_pretraining.py", line 506, in main model, optimizer, lr_scheduler, checkpoint, global_step, criterion = prepare_model_and_optimizer(args, device) File "/workspace/bert/run_pretraining.py", line 359, in prepare_model_and_optimizer args.resumestep = max([int(x.split('.pt')[0].split('')[1].strip()) for x in model_names]) ValueError: max() arg is an empty sequence Traceback (most recent call last): File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in main() File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main cmd=cmd) subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-u', '/workspace/bert/run_pretraining.py', '--local_rank=7', '--input_dir=/workspace/bert/data/hdf5_lower_case_1_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/', '--output_dir=/workspace/bert/results/checkpoints', '--config_file=bert_config.json', '--bert_model=bert-large-uncased', '--train_batch_size=4096', '--max_seq_length=512', '--max_predictions_per_seq=80', '--max_steps=1563', '--warmup_proportion=0.128', '--num_steps_per_checkpoint=200', '--learning_rate=4e-3', '--seed=12439', '--fp16', '--gradient_accumulation_steps=512', '--allreduce_post_accumulation', '--allreduce_post_accumulation_fp16', '--do_train', '--phase2', '--resume_from_checkpoint', '--phase1_end_step=7038', '--json-summary', '/workspace/bert/results/dllogger.json']' returned non-zero exit status 1.

  ---

  Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

  ---

  finished phase2

I have the following folder under data directory. (Noted that folder name hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5 is different from hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10 in the script output)

/workspace/bert# ls data BooksDownloader.py BookscorpusTextFormatting.py Downloader.py GLUEDownloader.py GooglePretrainedWeightDownloader.py NVIDIAPretrainedWeightDownloader.py SquadDownloader.py TextSharding.py WikiDownloader.py WikicorpusTextFormatting.py init.py pycache bertPrep.py create_datasets_from_start.sh download extracted formatted_one_article_per_line hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5 hdf5_lower_case_1_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5 sharded_training_shards_256_test_shards_256_fraction_0.1 squad

[sharathts]

Thank you for pointing this out (it is a name mismatch in run_pretraining.sh). This has been fixed in https://github.com/NVIDIA/DeepLearningExamples/pull/845

[jzhang82119]

The subfolder also need to be modified. The wikicorpus_en is the lowest level folder, there is no books_wiki_en_corpus/training/

this is what I have hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5/wikicorpus_en/

[sharathts]

Thank you. I have updated #845 to reflect the above fix as well.

[sharathts]

@jzhang82119 Also note that hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5/books_wiki_en_corpus is the default path assuming both wikipedia and bookcorpus have been downloaded.

It looks like you downloaded wiki only. For that, you can change books_wiki_en_corpus in the path name to wikicorpus_en.