Open minhdang241 opened 3 years ago
While we have built tokenizer for test set, the corresponding embeddings will be randomly initialized for words that do not exist in the pre-trained word embedding. That is, if a word is not available in the vocabulary of the pre-trained embedding, it will be randomly initialized.
I don't think we should build the tokenizer with both train and test set for non-BERT model situations. The chances are there will be words in the production environment which are not available in our vocabulary. If we use the test set to build the vocabulary, the performance will be bias.