search

google-research / albert

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Apache License 2.0
3.23k stars 570 forks source link

model isn't learning #207

Open raviolli opened 4 years ago

raviolli commented 4 years ago

Hi there, I have loaded up an albert model and loaded the weights just fine no errors or warning. but when I go to train it the loss never goes lower it stays around 0.7 and the accuracy stays around 0.5

I'm trying to perform a fine-tuning with a dense layer of size 2 classes. Any thoughts?

I've checked the data X and Y values and I think their correct. ids is tokenized+encoded. the mask is 1 for ids and 0 for padding. the type is 0 for all tokens. I'm using SpareCategoricalCrossEntropy so the label is 0 or 1 and not OHE.

I'm so stuck. Any help is greatly appreciated.

Danny-Google commented 4 years ago

Could you post more info like training info? Also, you may want to start with our colab tutorial.