Add JUnit back in - Githubissues

nohwnd commented 4 years ago

There is some work done towards that but the internal impelmentation and test need to be rewritten to P and to use the new structure.

fflaten commented 4 years ago

Anyone working on this yet?

bgelens commented 4 years ago

@nohwnd I contributed the original support but I'm not using junit myself anymore so I did not miss it :)

If you need help with this, you can assign this issue to me

fflaten commented 4 years ago

@nohwnd If you're not already working on this, I could give it a shot.

nohwnd commented 4 years ago

@fflaten I am not working on it. But I think I can give it a quick shot because I remember some of the stuff I needed to do for the NUnit output to work. If I don't succeed entirely you can then push to my branch if I add you appropriate permissions.

IIRC the JUnit output structure copies the NUnit code structure. I don't think there were many changes in the nunit part since it moved to v5. What I recall was that I needed to change the way the result tree is navigated, and add some early returns for filtered out tests and blocks. There was one problematic function where both test and block could be passed in, and there I needed to add a condition to decide how to move deeper into the tree.

It also helped a lot to run the same set of simple tests against v4 and compare the xml output in vscode, by opening both files and comparing them using the "Compare active file with..." command

fflaten commented 4 years ago

Compared it with some Pester4 XMLs earlier myself and remember a few thoughts:

Update parameter names (action and node -> block, testresult) to match nunit better
Copy the path-switch from nunit to get path/name for scriptblocks
Use ExpandedPath where needed
Since it doesn't care about contexts (nested test-suite are not allowed), maybe use Fold-Block to simplify current testcases-lookup.
Add stack trace as inner text at failure-element
Configuration object + Invoke-Pester help has typos. They list possible (and default) values as NUnit2.5 and not NUnitXml (and soon JUnitXml) which are the only one actually values allowed.

Also noticed that nunitxml uses the DisplayedErrorMessage-property for failuremessage. That causes file:line to be shown both in failuremessage and stack trace. Maybe strip out "at item:line" to avoid duplicate info the in test report?

nohwnd commented 4 years ago

https://stackoverflow.com/a/26661423/3065397 this is how we can add the stack trace, as text in the failure tag.

fflaten commented 4 years ago

Agreed

Add stack trace as inner text at failure-element

This suggestion wasn't luck, Actually tested in Azure DevOps earlier 🙂

<testcase name="describe state tests.Oh my god.passing with testcases 1" status="Failed" classname="/workspaces/Pester/demo.tests.ps1" assertions="0" time="0.020">
  <failure message="Expected 2, but got 1.&#xA;at 1 | Should -Be 2, /workspaces/Pester/demo.tests.ps1:18">at &lt;ScriptBlock&gt;, /workspaces/Pester/demo.tests.ps1:18</failure>
</testcase>

`

nohwnd commented 4 years ago

The JUnit tests we have in v4 seem to be wrong.

https://github.com/pester/Pester/blob/bb59f5bf274ea33d348aa5376496dc188583d5a4/Functions/TestResults.Tests.ps1#L311

This says that for 2 describes we should get two test suites, but if I run this in v4:

Describe "Mocked Describe" {
    It "Successful testcase" {
        $true | Should -Be $true
    }
    It "Failed testcase" {
        $false | Should -Be $true
    }
} 

Describe "Mocked Describe 2" {
    It "Successful testcase" {
        $true | Should -Be $true
    }
    It "Failed testcase" {
        $false | Should -Be $true
    }
}

It produces this xml (formatted):

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<testsuites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="junit_schema_4.xsd" name="Pester" tests="2" errors="0" failures="2" disabled="0" time="0.343">
  <testsuite name="C:\temp\junit\test1.tests.ps1" tests="4" errors="0" failures="2" hostname="NDESKTOP" id="0" skipped="0" disabled="0" package="C:\temp\junit\test1.tests.ps1" time="0.343">
    <properties>
      <property name="os-version" value="10.0.19042" />
      <property name="user-domain" value="EUROPE" />
      <property name="machine-name" value="NDESKTOP" />
      <property name="user" value="jajares" />
      <property name="platform" value="Microsoft Windows 10 Pro N|C:\WINDOWS|\Device\Harddisk0\Partition4" />
      <property name="clr-version" value="Unknown" />
      <property name="cwd" value="C:\temp\junit" />
      <property name="junit-version" value="4" />
    </properties>
    <testcase name="Mocked Describe.Successful testcase" status="Passed" classname="C:\temp\junit\test1.tests.ps1" assertions="0" time="0.005" />
    <testcase name="Mocked Describe.Failed testcase" status="Failed" classname="C:\temp\junit\test1.tests.ps1" assertions="0" time="0.013">
      <failure message="Expected $true, but got $false." />
    </testcase>
    <testcase name="Mocked Describe 2.Successful testcase" status="Passed" classname="C:\temp\junit\test1.tests.ps1" assertions="0" time="0.009" />
    <testcase name="Mocked Describe 2.Failed testcase" status="Failed" classname="C:\temp\junit\test1.tests.ps1" assertions="0" time="0.006">
      <failure message="Expected $true, but got $false." />
    </testcase>
  </testsuite>
</testsuites>

While the test produces this XML (formatted):

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<testsuites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="junit_schema_4.xsd" name="Pester" tests="1" errors="0" failures="1" disabled="0" time="0.000">
    <testsuite name="Describe #1" tests="1" errors="0" failures="0" hostname="NDESKTOP" id="0" skipped="0" disabled="0" package="Describe #1" time="0.201">
        <properties>
            <property name="machine-name" value="NDESKTOP" />
            <property name="cwd" value="C:\Projects\pester4" />
            <property name="clr-version" value="Unknown" />
            <property name="user-domain" value="EUROPE" />
            <property name="junit-version" value="4" />
            <property name="platform" value="Microsoft Windows 10 Pro N|C:\WINDOWS|\Device\Harddisk0\Partition4" />
            <property name="os-version" value="10.0.19042" />
            <property name="user" value="jajares" />
        </properties>
        <testcase name="Successful testcase" status="Passed" classname="Describe #1" assertions="0" time="0.201" />
    </testsuite>
    <testsuite name="Describe #2" tests="1" errors="0" failures="1" hostname="NDESKTOP" id="1" skipped="0" disabled="0" package="Describe #2" time="0.201">
        <properties>
            <property name="machine-name" value="NDESKTOP" />
            <property name="cwd" value="C:\Projects\pester4" />
            <property name="clr-version" value="Unknown" />
            <property name="user-domain" value="EUROPE" />
            <property name="junit-version" value="4" />
            <property name="platform" value="Microsoft Windows 10 Pro N|C:\WINDOWS|\Device\Harddisk0\Partition4" />
            <property name="os-version" value="10.0.19042" />
            <property name="user" value="jajares" />
        </properties>
        <testcase name="Failed testcase" status="Failed" classname="Describe #2" assertions="0" time="0.200">
            <failure message="" />
        </testcase>
    </testsuite>
</testsuites>

So imho the test are wrong and are using describes in places of containers (files).

I don't have much experience with JUnit but, I will target the actual output of the v4, to get the same in v5. Not what the broken tests say is correct.

nohwnd commented 4 years ago

$XmlWriter.WriteAttributeString('errors', '0')

should be FailedBlocks + FailedContainers

fflaten commented 4 years ago

So imho the test are wrong and are using describes in places of containers (files).

I don't have much experience with JUnit but, I will target the actual output of the v4, to get the same in v5. Not what the broken tests say is correct.

You're right, test is wrong. Limited experience with this, but played with JUnit 5 in a devcontainer. JUnit creates one test-suite per class in the root a file. Nested containers (e.g. nested classes in java) were implemented in JUnit 5, but they are also placed in the testsuite of the "root class" in the file. A $-delimiter is used to show subclass-structure, something Pester already does with . in expandedpath.

Ex.

class TestingAStackDemo {
    @Test
    void isInstantiatedWithNew() {
        //assert
    }

    @Nested
    class WhenNew {

        @Test
        void isEmpty() {
            //assert
        }

        @Nested
        class AfterPushing {

            @Test
            void isNotEmpty() {
                //assert
            }
        }
    }
}

Xml-report:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="TestingAStackDemo" time="0.032" tests="3" errors="0" skipped="0" failures="0">
  <properties>
    ...
  </properties>
  <testcase name="isInstantiatedWithNew" classname="TestingAStackDemo" time="0.008"/>
  <testcase name="isEmpty" classname="TestingAStackDemo$WhenNew" time="0.001"/>
  <testcase name="isNotEmpty" classname="TestingAStackDemo$WhenNew$AfterPushing" time="0.001"/>
</testsuite>

pester / Pester

Add JUnit back in #1515