issue with continueOnStepFailure and waitFor

[joelpramos]

As reported in https://stackoverflow.com/questions/70938149/is-there-any-way-to-use-continueonstepfailure-with-methods-waitfor-waitforurl

Issues present:

1. null pointer is triggered when a non-DSL keyword is used (e.g. 'waitFor')
2. no way to evaluate configure the skip when JS expressions are used inline (should support as 'eval' keyword is implicit)
3. when using multi line expressions the mechanism does not work

[joelpramos]

Just some food for thought here for me or others who wish to contribute: my perspective as I provided the fix is that waitFor is NOT part of the DSL. Taking the approach of purely reading and understanding of the Karate code / engine works behind the scenes this is true however reading the Karate UI documentation there are a lot of methods in there that could be considered as part of the DSL framework - waitFor(), click(), input(), etc.

The Karate implementation actually just has these methods as methods of the Driver behind use and so it's like for the test scripters these are indeed part of the DSL because that's how it is intended.

For now the fix supports all of this by taking an approach of fixing an issue where there was no way to skip inline javascript expressions (i.e. lines that are only javascript such as an if condition with brackets). These driver methods are evaluated as such and so these could be used as a workaround.

When I originally introduced this continueOnStepFailure in the framework the main intent was to allow for large blocks / set of lines doing match conditions to all be executed for those cases in which you have multiple failures in your assertions and you're just looking at getting as much info as possible to fix as many issues in your API as possible.

Rajeev's post on stackoverflow makes me realize others might be looking to use this to introduce conditions in the tests. I think Peter and I debated this for a while in some other tests and the consensus was that it's not ideal to have conditions in a test. That is way even after finishing the "block" of continueOnStepFailure and even if all the other tests of your script pass the Scenario will still be marked as a FAIL. Think of it as a fail-safe mechanism to make sure users don't use these keyword as a way to workaround to the lack of if blocks in Karate (a design choice which can also very easily be circumvented with if conditions and calling feature files).

At the end of the day I think for a test scripts the keywords in the Karate documentation should be taken as what's part of the DSL, not what or how the framework is coded behind the scenes. To that end I think the methods present in the Driver class should also be candidates to be available as part of the set of methods that the configuration of the continueOnStepFailure supports. One way or the other the design principle of avoiding the if-block is retained and we just can't foresee or predict all possibilities test scripts might need this mechanism.

Unfortunately I'm low on cycles so it'll have to wait a bit before I do that change but shouldn't be too that bad to implement. Hopefully I have time in the upcoming weeks.

@ptrthomas we had decided back then to keep this configuration out of the official docs but now I wonder whether it should be added with a reference to the original thread (https://github.com/karatelabs/karate/issues/1446) and this comment as it explains the scope and objective of the keyword.

[ptrthomas]

@joelpramos yes we can add docs. I propose opening a new folder in the project root called docs creating a new markdown file there and linking to it from the main README. over time we can start migrating some more sections out of the monolithic README. I also want to improve the JS API section etc.

[ptrthomas]

1.2.0 released