Custom error message on Match

[coxjaso]

I am looking for a custom error message if a match fails, but I don't see docs on how to do this. Trivial Example:

Given path '/channel/' + channelID And request {: } When method patch Then status 200 And match response.data.channel contains """ {

: } """, **"An error message that is detailed....."**

[ptrthomas]

this is not directly supported. the cucumber-html report suffices for most purposes and you can use cucumber comments / print statements to add more context if needed.

this should also give you some ideas:

* def a = 2
* eval if (a != 1) throw 'a is not equal to one'

[sumargoraymond]

Hi @ptrthomas I personally think this is quite an important feature to have on karate. I had graphql tests which validate response using match operators.

The actual failure may look like this:

{"errors":[{"message":"NotFound: media not found","path":["FindMediaByID"],"extensions":{"code":"NotFound","message":"media not found"}}],"data":{"FindMediaByID":null}}

I am doing assertion on karate like:

And match response.data.FindMediaByID.id != '#null'

The error thrown in karate will be this instead:

assertion failed: path: $.data.FindMediaByID.id, actual: null, NOT expected: '#null', reason: actual json-path does not exist

Since we utilize email report to send the test execution result, we need to expand the response string first before knowing the root cause of the error on the assertion.

[ptrthomas]

@sumargoraymond do this in an afterScenario hook, and extract the response. all good: https://github.com/intuit/karate#hooks

sorry, I'm not convinced. and if you really feel so strongly about this, kindly implement it yourself

[austenjt]

I agree with sumargoraymond. Here is an example, where i have a fast-fail step before my other test steps. Unfortunately, I cannot give it a human readable message that says "Test requires an initial doc status of Finalized". I have had other people ask me "why does it fail" but because i cannot tag it with a human message, i always have to explain

Steps 
  # 1. check that initial status of test case is 'Finalized', or fail fast
  * path global.meta_uri, "docUri"
  * header Content-Type = 'application/json'
  * header Authorization = bearer
  * header Ocp-Apim-Subscription-Key = subscription_key
  * request ""
  * params { v3DocId: "#(global.document_id)" }
  * method get
        -- Doc string debug output--
  * status 200
  * match response[0].status == 'Finalized'

    match
      match failed: EQUALS
        $ | not equal (STRING:STRING)
        'Pending Finalize'
        'Finalized'

[ptrthomas]

@austenjt agreed. am re-opening this - also there is a thought of implementing "soft assertions" for JSON since many teams have asked for this, so hopefully can do this in the same re-factor.

my proposal so as to not break the simplicity of the match syntax is to "pre configure" the error message. meaning that we introduce a convention that if there is a comment just before the match that failed, that comment will be used as the match failure description. for example:

    # the initial doc status should be 'Finalized'
    * match response[0].status == 'Finalized'

so the rule is that the line before should be a comment.

[Mike-c-Jackson]

Would putting the assertions/match in a conditional provide the desired effect as this is how we accomplish it today

* if ( responseStatus == 500 ) matchResult = karate.match("$ contains {txnId: '#(txnIdCheck)', timestamp: '#(timestampCheck)', code: 'SERVER_ERROR', message: 'SERVER_ERROR: Failed to get equipment definition.'}")
            * if (matchResult.pass == false) { karate.logger.debug('matchResult failed:', matchResult.message); }    
        * match matchResult.pass == true

or with a simple conditional

 * if ($.accountNumber == '') { karate.logger.debug('These calls cannot be ran for empty Accounts!'); karate.abort(); }

[ptrthomas]

@Mike-c-Jackson I think the new proposal is far simpler and readable than the conditional.

as part of this issue, we can have karate.abort('message') take an optional string message, that will simplify your second example.

[austenjt]

Q1: Why put alt message on karate.abort but not on karate.fail?

Q2: Which would you use in a test setup/background situation vs. on a test failure that you want to be "soft"?

[ptrthomas]

@austenjt karate.fail() already takes a message. in this ticket we are not considering soft assertions.