The accuracy from pretraining is worse than without pretraining

[Hu-Pen-Chih]

My current task is to classify the association between CVEs and CWEs. However, I've noticed that using BertModel.from_pretrained('bert-base-uncased') in the fine-tuning stage results in lower accuracy compared to when I pretrain with more CVE-related descriptions first, and then fine-tune using the pretrained model.pt. I don’t understand why this is happening as I have ruled out compatibility issues with the model. It’s worth mentioning that in the pretraining phase, I only use the pretrained model weights for fine-tuning, and the tokenizer is consistently BertTokenizer.from_pretrained('bert-base-uncased'). I did not retrain or expand the tokenizer during pretraining because it is very time-consuming.

Here are the hyperparameters I am using:

batch_size = 16
num_epochs = 10
learning_rate = 1e-4
eps=1e-8
beta1 = 0.9
beta2 = 0.99
weight_decay = 0.01
total_steps = num_epochs * len(train_loader)
warmup_steps = total_steps // 10
early_stopping_patience = 2

Additionally, the settings for masked language modeling (MLM) are:

mask_prob=0.15
replace_mask_prob=0.8
random_replace_prob=0.10
keep_original_prob=0.10

I hope someone can answer my question. If more detailed code is needed, I can provide it. Thank you.