search

CUMLSec / trex

MIT License
89 stars 13 forks source link

Problems in running preprocess.py #9

Open Jerr-boop opened 3 years ago

Jerr-boop commented 3 years ago

2021-06-21 08:56:05 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input0/op_pos_emb', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/op_pos_emb/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.op_pos_emb.input0', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.op_pos_emb.input0', wandb_project=None, workers=40) 2021-06-21 08:56:05 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input0/inst_pos_emb', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/inst_pos_emb/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.inst_pos_emb.input0', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.inst_pos_emb.input0', wandb_project=None, workers=40) 2021-06-21 08:56:05 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input0/arch_emb', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/arch_emb/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.arch_emb.input0', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.arch_emb.input0', wandb_project=None, workers=40) 2021-06-21 08:56:05 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input0/static', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/static/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.static.input0', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.static.input0', wandb_project=None, workers=40) Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main

File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args)main(args)main(args)

  File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main

File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary src_dict = task.load_dictionary(args.srcdict)

src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary

File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename)return Dictionary.load(filename)
return Dictionary.load(filename)
return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load

File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe raise fnfe

File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/inst_pos_emb/dict.txt' with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/arch_emb/dict.txt' with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/op_pos_emb/dict.txt' raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/static/dict.txt' 2021-06-21 08:56:12 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input1/inst_pos_emb', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/inst_pos_emb/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.inst_pos_emb.input1', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.inst_pos_emb.input1', wandb_project=None, workers=40) 2021-06-21 08:56:12 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input1/arch_emb', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/arch_emb/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.arch_emb.input1', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.arch_emb.input1', wandb_project=None, workers=40) Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/inst_pos_emb/dict.txt' with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/arch_emb/dict.txt' Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/op_pos_emb/dict.txt' 2021-06-21 08:56:12 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input1/op_pos_emb', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/op_pos_emb/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.op_pos_emb.input1', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.op_pos_emb.input1', wandb_project=None, workers=40) Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/static/dict.txt' 2021-06-21 08:56:12 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input1/static', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/static/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.static.input1', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.static.input1', wandb_project=None, workers=40) Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/byte4/dict.txt' 2021-06-21 08:56:19 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input0/byte4', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/byte4/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.byte4.input0', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.byte4.input0', wandb_project=None, workers=40) 2021-06-21 08:56:20 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input0/byte1', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/byte1/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.byte1.input0', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.byte1.input0', wandb_project=None, workers=40) Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/byte1/dict.txt' Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/byte2/dict.txt' 2021-06-21 08:56:20 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input0/byte2', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/byte2/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.byte2.input0', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.byte2.input0', wandb_project=None, workers=40) 2021-06-21 08:56:20 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input0/byte3', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/byte3/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.byte3.input0', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.byte3.input0', wandb_project=None, workers=40) Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/byte3/dict.txt' Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/byte2/dict.txt' 2021-06-21 08:56:26 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input1/byte2', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/byte2/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.byte2.input1', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.byte2.input1', wandb_project=None, workers=40) 2021-06-21 08:56:26 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input1/byte4', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/byte4/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.byte4.input1', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.byte4.input1', wandb_project=None, workers=40) 2021-06-21 08:56:26 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input1/byte1', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/byte1/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.byte1.input1', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.byte1.input1', wandb_project=None, workers=40) Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/byte4/dict.txt' Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/byte1/dict.txt' Traceback (most recent call last): File "E:\miniconda\envs\yolov5\Scripts\fairseq-preprocess-script.py", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 394, in cli_main main(args) File "C:\Users\12433\Desktop\trex\fairseq_cli\preprocess.py", line 98, in main src_dict = task.load_dictionary(args.srcdict) File "C:\Users\12433\Desktop\trex\fairseq\tasks\fairseq_task.py", line 98, in load_dictionary return Dictionary.load(filename) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 216, in load d.add_from_file(f) File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 229, in add_from_file raise fnfe File "C:\Users\12433\Desktop\trex\fairseq\data\dictionary.py", line 226, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: 'data-bin/pretrain_dict/byte3/dict.txt' 2021-06-21 08:56:26 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/similarity/input1/byte3', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=False, log_file=None, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=True, optimizer=None, padding_factor=8, plasma_path='/tmp/plasma', profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang=None, srcdict='data-bin/pretrain_dict/byte3/dict.txt', suppress_crashes=False, target_lang=None, task='translation', tensorboard_logdir=None, testpref=None, tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='data-src/similarity/train.byte3.input1', use_plasma_view=False, user_dir=None, validpref='data-src/similarity/valid.byte3.input1', wandb_project=None, workers=40) Traceback (most recent call last): File "C:/Users/12433/Desktop/trex/command/finetune/preprocess.py", line 37, in main() File "C:/Users/12433/Desktop/trex/command/finetune/preprocess.py", line 34, in main shutil.copy('data-src/similarity/train.label', directory) File "E:\miniconda\envs\yolov5\lib\shutil.py", line 409, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "E:\miniconda\envs\yolov5\lib\shutil.py", line 259, in copyfile with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst: FileNotFoundError: [Errno 2] No such file or directory: 'data-src/similarity/train.label'

can you tell me how to solve it?

ice-tong commented 3 years ago

According to the error log, you perhaps run the preprocess.py under a wrong work directory. correct way:

~# cd /path/to/trex
~/path/to/trex# python command/finetune/preprocess.py