karatelabs / karate

Test Automation Made Simple
https://karatelabs.github.io/karate
MIT License
8.33k stars 1.95k forks source link

Karate contains only shortcut fails in v1.4.1 #2515

Open mapwake opened 9 months ago

mapwake commented 9 months ago

According to Karate documentation for contains short-cuts The following match arguments should pass only for the first expectation:

 * def cat = 
  """
  {
    name: 'Billie',
    kittens: [
      { id: 23, name: 'Bob', bla: [{ b: '1'}] },
      { id: 42, name: 'Wild' }
    ]
  }
  """
# this will pass and would be my expectation
* def expected = [{ id: 42, name: 'Wild' }, { id: 23, name: 'Bob', bla: [{ b: '1'}]}]
* match cat == { name: 'Billie', kittens: '#(^^expected)' }

# this also pass but i would assume, that it does not pass
* def expected = [{ id: 42, name: 'Wild' }, { id: 23, name: 'Bob', bla: { b: '1'} }]
* match cat == { name: 'Billie', kittens: '#(^^expected)' }

The second one should fail, because the expectation for the parameter "bla" is an object and not an array with an object in it.

I'm using Karate v1.4.1. The same scenario does work as i would assume with v1.2.0.

ptrthomas commented 9 months ago

probably related to changes here, needs investigation https://github.com/karatelabs/karate/issues/2170

f-delahaye commented 9 months ago

That's an interesting one ...

My understanding is that:

There are probably a few steps I missed/misunderstood, so feel free to correct me. But hopefully, the general idea is the right one.

Basically, as soon as contains only is converted into contains only deep, there's no way back. If line 278 is commented so as to keep contains only, the test fails as OP expects. I suspect that if line 580 is changed so that contains only deep is reverted back to contains only, it would fail too. Obviously, neither change is possible as it would break backwards compatibility.

I can't think of any other way to fix the issue, but maybe someone else will have an idea...

ptrthomas commented 9 months ago

@f-delahaye yes this is a complicated one :|

f-delahaye commented 8 months ago

Actually ... there might be a solution.

Line 278 is key, IMO.

This method handles macros, which presumably are the same thing as shortcuts. So when ^^ (contains only) is specified, it gets converted to contains only deep. Is this by design? This does not sound like the expected behaviour.

@ptrthomas we could comment out line 278, so that nestedType remains contains only? The test then fails, as expected by the user. This would be a breaking change, but we could add a new shortcut for contains only deep, if needed.

With the line commented out, and using the standard syntax as opposed to the shortcuts: match cat contains only deep { name: 'Billie', kittens: [{ id: 42, name: 'Wild' }, { id: 23, name: 'Bob', bla: { b: '1'} }]} passes, but we did specify contains only deep so that sounds reasonable.