search

codeforequity-at / botium-core

The Selenium for Chatbots - Bots Testing Bots
https://www.botium.ai
MIT License
229 stars 64 forks source link

lex v2 connector, user sends incorrect sessionState from last round #674

Closed Gu-Tina closed 2 years ago

Gu-Tina commented 2 years ago

Describe the bug lex v2 connector, when 'UserSays', it inherits "bot" sessionState and sends it to the bot. This sometimes put the bot on the undesired state, since 'bot' sessionState has 'dialogAction' which isn't set by the user.

Also, can add the support to set 'SessionState' in the script? To Reproduce Write a script which starts with noises, then utterances which the bot can recognize. The bot firstly replies with 'Fallback' which is correct, then replies with 'no clarification prompt', which is incorrect.

The reason is that on the second round, 'UserSays' sends out the sessionState which is from first round bot response. Since the sessionState on 1st round has dialogAction, when the user sends it, it sometimes put the dialog on the undesired state.

Expected behavior The bot replies with recognition after 'Fallback'.

Botium Flavour: "botium-connector-lex": "^0.1.0", (latest) "botium-core 1.12.3

Additional context Proposed fix: Reuse only parts of sessionState from last round, not dialogAction. or minor update line 276 for UserSays sessionState: { sessionAttributes: this.sessionState ? this.sessionState.sessionAttributes : this.sessionAttributes },

Also, can add the support to set 'SessionState' in the script?

codeforequity-at commented 2 years ago

added to our backlog

codeforequity-at commented 2 years ago

So, after doing some research on this, IMHO Botium is handling the sessionState as intended by Lex - fiddling around with the sessionState is something that should not be done (except the sessionAttributes) - https://docs.aws.amazon.com/lexv2/latest/dg/using-sessions.html

If you want to have a new session and dump the previous sessionState, then in Botium this should be separate test cases instead of trying to do it in one test case. For each test case, Botium is starting a new Lex session.

It is already possible to manipulate the Lex session attributes with Botium - see here, but not the dialogState.

Gu-Tina commented 2 years ago

The issue is exactly because Botium is fiddling around with the sessionState, with previous bot's sessionState, not the sessionState desired by the user. The same scenario works on AWS its own UI test tool or the other tools.

Also it's ok for the user to set the sessionState, not limited to the sessionAttributes. thttps://docs.aws.amazon.com/lexv2/latest/dg/API_runtime_SessionState.html

codeforequity-at commented 2 years ago

As far as I can see, the sessionState is by default not touched at all by Botium - it starts with an empty one, and in UserSays it is always using the most recent sessionState returned from the previous API call.

I dont' know what the AWS UI test tool or the "other tools" are doing - do you have examples or docs how to better deal with the sessionState (instead of leaving it untouched) ?