search

RasaHQ / carbon-bot

Apache License 2.0
42 stars 31 forks source link

Use diet for entities and bump rasa up to 1.9.6 #27

Closed dakshvar22 closed 4 years ago

amn41 commented 4 years ago

how long does it take you to train this model?

dakshvar22 commented 4 years ago

@amn41 Took around 20 mins on a GPU. Haven't tried locally(CPU) but the training inside the GH action ran for 55 mins. Any idea why Run through test stories step fails?

dakshvar22 commented 4 years ago

Manually posting the cross-val results since there is a credentials issue on the last step of the GH action - 

2020-04-16 15:00:30 INFO     rasa.test  - CV evaluation (n=5)
2020-04-16 15:00:30 INFO     rasa.test  - Intent evaluation results
2020-04-16 15:00:30 INFO     rasa.nlu.test  - train Accuracy: 0.997 (0.001)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - train F1-score: 0.997 (0.001)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - train Precision: 0.997 (0.001)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - test Accuracy: 0.833 (0.017)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - test F1-score: 0.831 (0.017)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - test Precision: 0.839 (0.014)
2020-04-16 15:00:30 INFO     rasa.test  - Entity evaluation results
2020-04-16 15:00:30 INFO     rasa.nlu.test  - Entity extractor: DIETClassifier
2020-04-16 15:00:30 INFO     rasa.nlu.test  - train Accuracy: 0.999 (0.001)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - train F1-score: 0.994 (0.004)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - train Precision: 0.991 (0.009)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - Entity extractor: DIETClassifier
2020-04-16 15:00:30 INFO     rasa.nlu.test  - test Accuracy: 0.981 (0.005)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - test F1-score: 0.860 (0.025)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - test Precision: 0.847 (0.030)
2020-04-16 15:00:30 INFO     rasa.test  - Response Selection evaluation results
2020-04-16 15:00:30 INFO     rasa.nlu.test  - train Accuracy: 0.997 (0.001)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - train F1-score: 0.997 (0.001)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - train Precision: 0.997 (0.001)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - test Accuracy: 0.675 (0.028)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - test F1-score: 0.660 (0.030)
2020-04-16 15:00:30 INFO     rasa.nlu.test  - test Precision: 0.683 (0.039)
amn41 commented 4 years ago

thanks @dakshvar22 ! the error is probably bc the cross-val takes longer than an hour and the token expires