XSpec test fails but Maven process reports success

[galtm]

I have a case where an <x:expect> element in the XSpec test produces a failure, but the overall Maven process reports success. Here is a small test that reproduces the issue. (The XSLT stylesheet can be anything because it's not actually used.)

<?xml version="1.0" encoding="UTF-8"?>
<x:description xmlns:x="http://www.jenitennison.com/xslt/xspec"
         stylesheet="something.xsl" xslt-version="3.0">

  <x:scenario shared="yes" label="shared passing assertion">
    <x:expect label="passing assertion" test="true()"/>
  </x:scenario>

  <x:scenario label="Scenario with failing assertion that Maven plugin misses">
    <x:call function="string">
      <x:param select="'a'"/>
    </x:call>
    <x:expect label="failing assertion" select="'b'"/>
  </x:scenario>  

  <x:scenario label="Scenario that references shared assertion more than once">
    <x:scenario label="case 1">
      <x:call function="string">
        <x:param select="'a'"/>
      </x:call>
      <x:like label="shared passing assertion"/>
    </x:scenario>
    <x:scenario label="case 2">
      <x:call function="string">
        <x:param select="'z'"/>
      </x:call>
      <x:like label="shared passing assertion"/>
    </x:scenario>
  </x:scenario>

</x:description>

After I do mvn test the result includes the following.

9810 [INFO] Executing XSpec: repro-steps.xspec.xslt
Testing with SAXON EE 9.9.1.2
Scenario with failing assertion that Maven plugin misses
failing assertion
      FAILED
Scenario that references shared assertion more than once
..case 1
passing assertion
..case 2
passing assertion
passed: 2 / pending: 0 / failed: 1 / total: 3
9914 [INFO] repro-steps.xspec results [Passed/Pending/Failed/Missed/Total] = [2/0/1/-1/2]
9929 [INFO] ------------------------------------------------------------------------
9929 [INFO] BUILD SUCCESS
9929 [INFO] ------------------------------------------------------------------------

The problem seems to be related to code reuse in the XSpec test via the <x:like> element. If I delete one of the <x:like> elements or replace it with the content of the shared scenario, the problem goes away.

[AirQuick]

XSpecCounterCH seems to implement a simple XSpec document parser. I'm afraid it is not viable, because

• Resolving everything in an XSpec document and interpreting it is not that simple.
• The parser has to keep catching up bug fixes and specification changes.

[cmarchand]

Yes, it's an old heritage. Actually, I didn't know <x:like />.... This class was introduced to keep a viable result even if the XSpec crashes, for example if Xslt throws an exception, to be able to produce a report.

[cmarchand]

@galtm : I've added two test cases to support shared scenarios. Could you please check if the test cases are correct in their usage and expectations ?

Test 1

Test 2

Thanks in advance

[galtm]

Hi, @cmarchand .

Yes, the usage and expectations look good. I'm not familiar with the code in this repository, so I can't comment on the tests' connection with this issue or whether other test cases would be useful. I'll leave that part up to you. Thanks for working on this issue!

[galtm]

One more thing, in case it affects your fix or tests: The shared scenario is not limited to a single x:expect child. It can have zero or more x:expect elements and other kinds of elements as well.

[cmarchand]

Ok, I'll complete the test cases with this.