search

bryanlimy / tf2-transformer-chatbot

Transformer Chatbot in TensorFlow 2 with TPU support.
MIT License
130 stars 58 forks source link

What's the ideal Loss and Accuracy? #5

Closed thomasyue closed 5 years ago

thomasyue commented 5 years ago

Hi!

I'm new to chatbot and nlp, but I'm curious what's an ideal loss and accuracy? I'm training it on my own corpus which contains 21k questions and 21k answers.

num_layers=3, num_units=512, d_model=256.

Loss decreases very fast, but accuracy increases very slow. Is it overfitting? my current loss is 0.9 and acc is 0.06

arthurflor23 commented 5 years ago

Same here, it's that correct?

bryanlimy commented 5 years ago

This really depends on the dataset you're training on, as well as the level of variation you want your output sentence to be. Since this is a chatbot, we don't want the model to output the same reply every time given the same input. For this particular dataset, I am getting around 0.7-0.8 val_loss and 0.12-0.15 val_acc where I think the outputs of the chatbot are acceptable.