Azure / simdem

Tool for Simulating Demo's, delivering Tutorials and using documentation as tests.
MIT License
34 stars 17 forks source link

Incomplete results being captured in response #132

Open rgardler-msft opened 2 years ago

rgardler-msft commented 2 years ago

It looks like incomplete outputs are being captured by the REPL system. This results in incorrect similarity ratings being calculated. This has been worked around in a number of places by artificially reducing the expected results similarity value. For example:

This can be reproduced easily in simdem/demo_scripts/test/environment_test.md document by adding the expected similarity rating here:

` ` `bash
cat ../env.json
` ` `

Results:

` ` `expected_similarity = 0.9
{
  "PARENT_TEST": "Hello from the parent"
}
` ` `

This results in the following test failure output (debug = true):

FAILED

Similarity ratio:    0.4
Expected Similarity: 0.9

=============================

Expected results:

{
  "PARENT_TEST": "Hello from the parent"
}

Actual results:

 the parent"
}