Closed PhilippMarquardt closed 4 years ago
@PhilippMarquardt - it might be, that your learning rate is too high. Try different learning rates, i.e. somewhere around 1e-5, or check the examples under https://github.com/kpe/bert-for-tf2/tree/master/examples
Any luck with this. I set the LR to 5e-5 but nothing... same problem
I have the same issue with outputs never evolving away from 0.5 and loss remaining at 0.7 regardless of learning rate or optimizer. Could you confirm that it is working and post an example if possible?
I am having the same issue. The model is predicting everything as 1 class.
Hi! I tried finetuning the albert base/large model on the MovieReview dataset that is used in the bert example.
The model is created like this: def create_model(max_seq_len):
I've used
from bert.tokenization.albert_tokenization import FullTokenizer
for tokenizationEverything else is like in the provided bert example.
When executing the training loop, the accuracy stays at 50% and the loss doesn't really change from .7
Has anybody successfully finetuned any pretrained albert model on the MovieReview dataset? If yes, what am I doing wrong? Thanks in advance!