ZhaoQianfeng
commented
1 year ago @oraby8 Hi , I think the problem is with the '--arch' option, you need to make sure it matches your pre-trained nmt model.
I found that your pre-trained model's architecture is different from wmt19 winner model, for example, your model's encoder_embed_dim is 768 and wmt19 winner model's encoder_embed_dim is 1024. So you can't simply use --arch vanilla_knn_mt@transformer_wmt19_de_en when building datastore and inferencing.The correct steps are as follows:
-
add your arch function at the end of
knnbox/common_utils/archs.py.def my_arch(args): args.encoder_embed_dim = getattr(args, "encoder_embed_dim", 768) args.decoder_embed_dim = getattr(args, "decoder_embed_dim", 768) # other options ... base_architecture(args) -
regist your arch at the end of
knnbox/models/vanilla_knn_mt.py@register_model_architecture("vanilla_knn_mt", "vanilla_knn_mt@my_arch") def transformer_my_arch(args): archs.my_arch(args) -
use
--arch vanilla_knn_mt@my_archwhen build_datastore and inference.CUDA_VISIBLE_DEVICES=1 python $PROJECT_PATH/knnbox-scripts/common/validate.py $DATA_PATH \ --task translation \ --path $BASE_MODEL \ --source-lang en --target-lang ar \ --model-overrides "{'eval_bleu': False, 'required_seq_len_multiple':1, 'load_alignments': False}" \ --dataset-impl mmap \ --valid-subset train \ --skip-invalid-size-inputs-valid-test \ --max-tokens 2048 \ --bpe fastbpe \ --user-dir $PROJECT_PATH/knnbox/models \ --arch vanilla_knn_mt@my_arch \ --knn-mode build_datastore \ --knn-datastore-path $DATASTORE_SAVE_PATH \And there is no need to specify
encoder_embed_dimwhen inference, because--arch vanilla_knn_mt@my_archcontains this information:CUDA_VISIBLE_DEVICES=1 python $PROJECT_PATH/knnbox-scripts/common/generate.py $DATA_PATH \ --task translation \ --path $BASE_MODEL \ --dataset-impl mmap \ --beam 4 --lenpen 0.6 --max-len-a 1.2 --max-len-b 10 --source-lang en --target-lang ar \ --gen-subset test \ --max-tokens 2048 \ --scoring sacrebleu \ --tokenizer moses \ --remove-bpe \ --user-dir $PROJECT_PATH/knnbox/models \ --arch vanilla_knn_mt@my_arch \ --knn-mode inference \ --knn-datastore-path $DATASTORE_LOAD_PATH \ --knn-k 8 \ --knn-lambda 0.7 \ --knn-temperature 10.0 \
oraby8
when i test the base model it gives me 32.5 in blue score, when i do use vanilla-knn it always 0
to build datastore i used this configuration
CUDA_VISIBLE_DEVICES=1 python $PROJECT_PATH/knnbox-scripts/common/validate.py $DATA_PATH \ --task translation \ --path $BASE_MODEL \ --source-lang en --target-lang ar \ --model-overrides "{'eval_bleu': False, 'required_seq_len_multiple':1, 'load_alignments': False}" \ --dataset-impl mmap \ --valid-subset train \ --skip-invalid-size-inputs-valid-test \ --max-tokens 2048 \ --bpe fastbpe \ --user-dir $PROJECT_PATH/knnbox/models \ --arch vanilla_knn_mt@transformer_wmt19_de_en \ --knn-mode build_datastore \ --knn-datastore-path $DATASTORE_SAVE_PATH \and to test the the model i used this configurations
CUDA_VISIBLE_DEVICES=1 python $PROJECT_PATH/knnbox-scripts/common/generate.py $DATA_PATH \ --task translation \ --path $BASE_MODEL \ --dataset-impl mmap \ --beam 4 --lenpen 0.6 --max-len-a 1.2 --max-len-b 10 --source-lang en --target-lang ar \ --gen-subset test \ --max-tokens 2048 \ --encoder-embed-dim 768 \ --decoder-embed-dim 768 \ --dropout 0.2 \ --attention-dropout 0.0 \ --encoder-layerdrop 0 \ --decoder-layerdrop 0 \ --encoder-ffn-embed-dim 2048 \ --decoder-ffn-embed-dim 2048 \ --scoring sacrebleu \ --tokenizer moses \ --remove-bpe \ --user-dir $PROJECT_PATH/knnbox/models \ --arch vanilla_knn_mt@transformer_wmt19_de_en \ --knn-mode inference \ --knn-datastore-path $DATASTORE_LOAD_PATH \ --knn-k 8 \ --knn-lambda 0.7 \ --knn-temperature 10.0 \