When running multiple tests and the first test fails, then all other tests also fail

[Elias011]

Describe the bug When running multiple tests and the first test fails, then all other tests also fail.

To Reproduce

• In a folder have multiple test files (We used excel format)
• Let the first test fail intentionally
• Run those tests using Botium-CLI botium-cli run mochawesome --config .\botium.json --convos Tests/
• The first test will fail
• The other tests will also fail

Expected behavior Only the first test fails and the rest succeed

Screenshots and Log files BotiumTestResultLog.txt BotiumVerboseLog.txt

Botium Flavour: Botium CLI

Botium Version 1.0.3

Additional context We ran botium-cli using the docker image on docker hub. But we could also reproduce this problem with a locally installed botium-cli. Our current workaround is running botium-cli docker image version 0.0.77, where the issue is not present.

We use Botium to test a Chatbot running on Microsoft Bot Framework using the DirectLine connector.

[codeforequity-at]

I suspect this to be the reason why test cases are failing:

  DIRECTLINE3_ACTIVITY_TEMPLATE: {
    from: { name: 'Botium', id: '00000000-0000-0000-0000-000000000000' },
    locale: 'nl-NL',
    channelId: 'directline'
  },

All test cases are using the same user id, so the conversations can overlap if there are only minor problems with the timing.

I suggest to remove the id-attribute there and let Botium generate a unique id for each test run - if this is possible for your Bot Framework implementation.

[Elias011]

Thanks for your quick reaction:)

We removed the id attribute. Unfortunately, we still have the same issue. We checked the conversation id of each test in the verbose log, they are different from each other.

Here you can find the logs files: BotiumVerboseLogWithoutIdInConfig.txt BotiumTestResultLogWithoutIdInConfig.txt

[codeforequity-at]

I cannot see any real trouble in the log files.

In the test case there seems to be a wrong ".2" suffix for the "Hoi" conversation step:

"Hoi, met de chatbot van SnelStart." expected to match "Hoi, met de chatbot van SnelStart.2"

You can try to set the capability "DIRECTLINE3_GENERATE_USERNAME" to true, at this is actually required to make Botium generate unique ids as username - let's see if this helps.

[Elias011]

We tried your solution, but it did not generate an unique id for each test file. So the problem is still present. We added the ".2" suffix to reproduce the problem of having the first test fail, which then causes the other test running after that test to also fail. We noticed in the logs that the second test is trying to assert a reply from the first test case (see screenshot)

Also, this behavior does not happen in version 0.077.

BotiumVerboseLogWithGeneratedIdConfig.txt BotiumTestResultLogWithGeneratedIdConfig.txt

[codeforequity-at]

Can you please try to set the "DIRECTLINE3_WEBSOCKET" capability to false and try again

[Elias011]

We just tried and it worked! Thank you very much for the help 🙂