Closed oathaha closed 2 years ago
Follow this.
bart = BARTModel.from_pretrained(
args.checkpoint_dir,
checkpoint_file=args.checkpoint_file,
data_name_or_path=args.data_name_or_path,
user_dir=root_dir.joinpath('source'),
task=args.task,
bpe='sentencepiece',
sentencepiece_model=spm_dir,
)
The last 4 parameters are optional. But data_name_or_path
is crucial for dictionary loading. args.data_name_or_path
should contain the *.dict.txt
files.
I found that the vocab size of embedding layer is 50,004. However, the vocab size of subword (bpe) tokenizer is 50,044, which causes out-of-vocab problem.
I got the vocab size of tokenizer by using this code
vocab_size = len(bart.task.source_dictionary)
I faced this problem when I use
bart.encode()
function.Here is how I load the pre-trained bart model in python
bart = BARTModel.from_pretrained(model_path, checkpoint_file=model_file)
where model_path contains the pre-trained PLBART, and
model_file
isplbart_base.pt
I am not sure if I do something wrong here. Can anyone help me?
Thanks.