💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants
A customer has a bot working as expected with Rasa 2.3.3 and Rasa 2.6.1. For those versions their stories follow the AugmentedMemoizationPolicy. While upgrading to Rasa 2.8.7 they noticed inconsistency within the behavior of the AugmentedMemoizationPolicy where for the same stories they were using previously the AugMemo policy was now losing track of the conversation and deferring to the TEDPolicy to make a prediction, which they observed was largely a low confidence prediction.
Potential Source of the Problem
A potential explanation for this is because of the change introduced in Rasa 2.8 where the DeLorean phase of AugMemo began to trim the tracker history according to the max_history prior to constructing the prediction states. Previously the entire tracker history was being used until this commit applying the tracker trimming.
When the AugmentedMemoizationPolicy is trained all the states contain a component detailing the last user intent, like this: {user: {"intent": "x"}, in addition to the various other action and slot events.
For the customer's case their story has a bunch of sequential actions occurring between user utterances. Once the number of sequential actions exceeds max_history when the DeLorean trims the tracker the last user utterance is lost. So, when the DeLorean constructs the prediction states from the trimmed tracker the last user intent is not included in those states. As a result AugMemo will not find a match because the states constructed during training and at prediction time do not contain the same information.
Reproducing the Error
The problem can be reproduced with the repo linked in this slack discussion, which contains a similar bot to what the customer is using. You can train a model and follow this conversation:
User: Hi
Bot: What can I help you with?
User: /technical_support
The bot will process actions up to action_set_customer_service_path where the bot will no longer be able to follow the AugmentedMemoizationPolicy to make predictions. Instead TED makes a low confidence prediction. The expectation is the bot would follow the checkpoint to predict action_check_customer_service_path. Reviewing the construction of the mcfly_tracker after action_set_customer_service_path the trimmed tracker no longer contains the last user utterance because there are now 6 actions occurring after the utterance and the utterance gets trimmed.
kedz
commented
2 years ago
Rasa Open Source version
2.8.7
Rasa SDK version
No response
Rasa X version
No response
Python version
3.8
What operating system are you using?
OSX
What happened?
A customer has a bot working as expected with Rasa 2.3.3 and Rasa 2.6.1. For those versions their stories follow the
AugmentedMemoizationPolicy. While upgrading to Rasa 2.8.7 they noticed inconsistency within the behavior of theAugmentedMemoizationPolicywhere for the same stories they were using previously the AugMemo policy was now losing track of the conversation and deferring to theTEDPolicyto make a prediction, which they observed was largely a low confidence prediction.Potential Source of the Problem
A potential explanation for this is because of the change introduced in Rasa 2.8 where the DeLorean phase of AugMemo began to trim the tracker history according to the
max_historyprior to constructing the prediction states. Previously the entire tracker history was being used until this commit applying the tracker trimming.When the
AugmentedMemoizationPolicyis trained all the states contain a component detailing the last user intent, like this:{user: {"intent": "x"}, in addition to the various other action and slot events.For the customer's case their story has a bunch of sequential actions occurring between user utterances. Once the number of sequential actions exceeds
max_historywhen the DeLorean trims the tracker the last user utterance is lost. So, when the DeLorean constructs the prediction states from the trimmed tracker the last user intent is not included in those states. As a result AugMemo will not find a match because the states constructed during training and at prediction time do not contain the same information.Reproducing the Error
The problem can be reproduced with the repo linked in this slack discussion, which contains a similar bot to what the customer is using. You can train a model and follow this conversation:
The bot will process actions up to
action_set_customer_service_pathwhere the bot will no longer be able to follow theAugmentedMemoizationPolicyto make predictions. Instead TED makes a low confidence prediction. The expectation is the bot would follow the checkpoint to predictaction_check_customer_service_path. Reviewing the construction of themcfly_trackerafteraction_set_customer_service_paththe trimmed tracker no longer contains the last user utterance because there are now 6 actions occurring after the utterance and the utterance gets trimmed.Command / Request
No response
Relevant log output
No response