search

MahmoudWahdan / dialog-nlu

Tensorflow and Keras implementation of the state of the art researches in Dialog System NLU
Apache License 2.0
98 stars 40 forks source link

Way lower accuracy than example notebooks #41

Open J0ekr opened 1 year ago

J0ekr commented 1 year ago

Hey, Thanks for the great repo! I am currently trying to train a NLU and tested out your using_bert_crf_nlu example notebook with your provided data. I have tested both Bert and Albert embeddings. However, I always only get 14% accuracy (so random classification), while you have ~98% accuracy. I am currently using python 3.11 - the only major compatibility issue I had was with sentencepiece (dependency from transformers), where I now use the newest version instead of your specified version range. I can't imagine this being the case for the model not working anymore. I will continue investigating this, but wanted to ask whether you or someone else already had this issue and has ideas!

Thanks!

J0ekr commented 1 year ago

You set the BERT Embedding trainable=True. I have now set this to False, and get accuracies of ~80% after 3 epochs .

I assume you hotloaded your model used in the example notebook?