search

RasaHQ / rasa

💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants
https://rasa.com/docs/rasa/
Apache License 2.0
18.81k stars 4.62k forks source link

`rasa test` skips stories that `rasa test core` does not skip #7677

Closed koernerfelicia closed 2 years ago

koernerfelicia commented 3 years ago

Rasa version: 2.2.0

Python version: 3.7.2

Operating system (windows, osx, ...): OSX

Issue: rasa test ignores stories which rasa test core does not ignore.

Steps to reproduce Run rasa test in examples/e2ebot. 0 stories are evaluated. Run rasa test core in examples/e2ebot. 4 stories are evaluated. Expected: rasa test evaluated 4 stories.

Same thing happened with examples/moodbot

Error (including full traceback):

rasa test /Users/fkr/Repositories/rasa/rasa/shared/utils/io.py:93: UserWarning: The path 'tests' does not seem to exist. Using the default value 'data' instead. 2021-01-05 11:16:07 INFO rasa.model - Loading model models/20201221-142641.tar.gz... 2021-01-05 11:16:14 INFO rasa.core.policies.ensemble - MappingPolicy not included in policy ensemble. Default intents 'restart and back will not trigger actions 'action_restart' and 'action_back'. 2021-01-05 11:16:14 INFO rasa.core.test - Evaluating 0 stories Progress: 0it [00:00, ?it/s] 2021-01-05 11:16:14 INFO rasa.core.test - Finished collecting predictions. 2021-01-05 11:16:14 WARNING rasa.test - No labels to evaluate. Skip evaluation. 2021-01-05 11:16:14 INFO rasa.core.test - Evaluation Results on END-TO-END level: 2021-01-05 11:16:14 INFO rasa.core.test - Correct: 0 / 0 2021-01-05 11:16:14 INFO rasa.core.test - F1-Score: 0.000 2021-01-05 11:16:14 INFO rasa.core.test - Precision: 0.000 2021-01-05 11:16:14 INFO rasa.core.test - Accuracy: 0.000 2021-01-05 11:16:14 INFO rasa.core.test - In-data fraction: 0 2021-01-05 11:16:14 WARNING rasa.test - No labels to evaluate. Skip evaluation. 2021-01-05 11:16:14 INFO rasa.core.test - Stories report saved to results/story_report.json. 2021-01-05 11:16:14 INFO rasa.core.test - Evaluation Results on ACTION level: 2021-01-05 11:16:14 INFO rasa.core.test - Correct: 0 / 0 2021-01-05 11:16:14 INFO rasa.core.test - F1-Score: 0.000 2021-01-05 11:16:14 INFO rasa.core.test - Precision: 0.000 2021-01-05 11:16:14 INFO rasa.core.test - Accuracy: 0.000 2021-01-05 11:16:14 INFO rasa.core.test - In-data fraction: 0 /Users/fkr/Repositories/rasa/rasa/utils/plotting.py:80: UserWarning: Attempting to set identical left == right == -0.5 results in singular transformations; automatically expanding. norm=LogNorm(vmin=zmin, vmax=zmax), /Users/fkr/Repositories/rasa/rasa/utils/plotting.py:80: UserWarning: Attempting to set identical bottom == top == -0.5 results in singular transformations; automatically expanding. norm=LogNorm(vmin=zmin, vmax=zmax), 2021-01-05 11:16:14 INFO rasa.utils.plotting - Confusion matrix, without normalization: [] Failed stories written to 'results/failed_test_stories.yml' 2021-01-05 11:16:15 INFO rasa.model - Loading model models/20201221-142641.tar.gz... 2021-01-05 11:16:16 INFO rasa.nlu.test - Running model for predictions: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [00:00<00:00, 180.10it/s] 2021-01-05 11:16:16 INFO rasa.nlu.test - Intent evaluation results: 2021-01-05 11:16:16 INFO rasa.nlu.test - Intent Evaluation: Only considering those 33 examples that have a defined intent out of 33 examples. 2021-01-05 11:16:16 INFO rasa.nlu.test - Classification report saved to results/intent_report.json. 2021-01-05 11:16:16 INFO rasa.nlu.test - Every intent was predicted correctly by the model. 2021-01-05 11:16:16 INFO rasa.utils.plotting - Confusion matrix, without normalization: [[ 7 0 0 0] [ 0 7 0 0] [ 0 0 13 0] [ 0 0 0 6]] /Users/fkr/Repositories/rasa/rasa/utils/plotting.py:176: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding. axes[1].set(yticks=bins, xlim=(0, max_xlims[1]), ylim=(min_ylim, 1.0)) 2021-01-05 11:16:17 INFO rasa.nlu.test - Entity evaluation results:

vs:

$ rasa test core 2021-01-05 11:16:27 INFO rasa.model - Loading model models/20201221-142641.tar.gz... 2021-01-05 11:16:34 INFO rasa.core.policies.ensemble - MappingPolicy not included in policy ensemble. Default intents 'restart and back will not trigger actions 'action_restart' and 'action_back'. Processed story blocks: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 00:00<00:00, 1559.51it/s, # trackers=1 2021-01-05 11:16:34 INFO rasa.core.test - Evaluating 4 stories Progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 20.82it/s] 2021-01-05 11:16:34 INFO rasa.core.test - Finished collecting predictions. 2021-01-05 11:16:34 INFO rasa.core.test - Evaluation Results on CONVERSATION level: 2021-01-05 11:16:34 INFO rasa.core.test - Correct: 4 / 4 2021-01-05 11:16:34 INFO rasa.core.test - F1-Score: 1.000 2021-01-05 11:16:34 INFO rasa.core.test - Precision: 1.000 2021-01-05 11:16:34 INFO rasa.core.test - Accuracy: 1.000 2021-01-05 11:16:34 INFO rasa.core.test - In-data fraction: 0 2021-01-05 11:16:34 INFO rasa.core.test - Stories report saved to results/story_report.json. 2021-01-05 11:16:34 INFO rasa.core.test - Evaluation Results on ACTION level: 2021-01-05 11:16:34 INFO rasa.core.test - Correct: 24 / 24 2021-01-05 11:16:34 INFO rasa.core.test - F1-Score: 1.000 2021-01-05 11:16:34 INFO rasa.core.test - Precision: 1.000 2021-01-05 11:16:34 INFO rasa.core.test - Accuracy: 1.000 2021-01-05 11:16:34 INFO rasa.core.test - In-data fraction: 0 2021-01-05 11:16:34 INFO rasa.utils.plotting - Confusion matrix, without normalization: [[12 0 0 0] [ 0 2 0 0] [ 0 0 3 0] [ 0 0 0 7]] Failed stories written to 'results/failed_test_stories.yml'

Command or request that led to error:

rasa test

stale[bot] commented 3 years ago

Exalate commented:

stale[bot] commented:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.