results xml-file in nunit2 format does not confirm to xsd

[ChrisMaddock]

@jjvdgeer commented on Thu Aug 16 2018

When running:

nunit3-console.exe -result TestResults.xml;format=nunit2 SomeTest.dll SomeOtherTest.dll

it seems that the TestResults.xml is generating does not adhire to the xsd (as found here) I get a Result.xml which looks like this:

<test-results ...>
  <environment .../>
  <culture-info .../>
  <test-suite type="Assembly" name="SomeTest.dll" .../>
  <test-suite type="Assembly" name="SomeOtherTest.dll" .../>
</test-results>

However, according to the xsd, it's not allowed to have more than one test-suite element within the test-results:

<xs:complexType name="resultType">
    <xs:sequence>
        <xs:element name="environment" type="environmentType" />
        <xs:element name="culture-info" type="culture-infoType" />
        <xs:element name="test-suite" type="test-suiteType" />
    </xs:sequence>
    ...
</xs:complexType>

This is causing me problems as the xunit plugin for Jenkins rejects these results-files.

This is using NUnit-Console 3.8.0.

---

@ChrisMaddock commented on Thu Aug 16 2018

Hi @jjvdgeer,

The nunit2 result writing is handled by an extension to the engine. I'll move this issue over to the repository where it lives, and comment further there! 🙂

[ChrisMaddock]

Correction: I won't be commenting further, as the answer I thought I had was wrong - sorry! I'll leave this to people more familiar with NUnit 2 to comment on.

[CharliePoole]

@jjvdgeer What version of the console runner are you using and what package did you install? If you installed the v2 result writer extension separately, what version is it?

[jjvdgeer]

@CharliePoole Its the NUnit.Console-3.8.0.msi I installed. I didn't install the v2 result writer extension separately.

[CharliePoole]

@jjvdgeer Thanks, that's what we needed. 😄

[CharliePoole]

Well, my question was academic, since the extension has not been changed for two years!

Apparently, when using it with more than one assembly on the command line it has always done what you are seeing. I believe NUnit V2 created an anonymous project to wrap multiple assemblies and would therefore only have a single initial suite. The adapter should probably do the same when there are multiple assemblies listed under the test-run element.

Possible workarounds:

1. Run one assembly at a time
2. Create an NUnit project to wrap multiple assemblies

[jjvdgeer]

These workarounds would take some time to implement and that's a problem. However, the xunit plugin team has done a fix that should solve the problem I think. Haven't tested it yet though.

[nfalco79]

I relax the constraint on the original XSD copy on xunit-plugin project but this solve only temporarily because on next release of NUnit this will be broken again.

[nfalco79]

To respect the XSD v2.6 in NUnit2XmlResultWriter#WriteXmlOutput after InitializeXmlFile should create a parent element test-suits (I do know which should be the correct type) for all test-suites children of type 'Assembly'.

This because for the XSD under resultType element only one test-suite element is permitted.

[CharliePoole]

I'm still wondering why this is suddenly a problem when it has always worked the same way as far as I can see.

[nfalco79]

It works because there are no test cases that verify if the generated report is NUnit 2.6 compliant or no one has opened a bug before.

The sense of releasing an XSD file with the distribution package is to provide a verification tool.

[CharliePoole]

@nfalco79 Sure, that's clear. What's not is how the output is suddenly not accepted in Jenkins. I would imagine this is a common use case. When a non-issue suddenly becomes an issue, we should try to figure out how it came about.

Currently, nobody is maintaining this extension. We discussed some options recently, but without a conclusion. Understanding how this arose at help with making a decision.

Finally, there's the question of other consumer programs. Have they adapted to the incorrect output? If so how will they be affected?

Hopefully you can see that, at this point, it's not quite so simple as just making it conform to the XSD would imply.

[nfalco79]

The jenkins plugin takes NUnit reports (more types are supported) and performs an XSLT to transform each NUnit report into a JUnit report. These reports are fed to junit plugin to build a summary. To performs the XSLT I have to base logic on a precise and fixed format. There was open issues about XSLT failures and usually the main reason was because some of the NUnit reports reported was generated by tools that did produce a similar NUnit report. After spend times to understand what was wrong or correct every report I have decided to adopt XSD (if one is defined) for any report tools supported. In this case the process is stopped before perform the XSLT, warning the user about the wrong format. The goal is to discard the issues on reports not generated correctly by the unofficial tools and in case, report an issue to the tool.

I expect a fix to XSD or to the tool to cover the case of multiple assemblies.

To be honest this is not the unique changes to the XSD. Was reported to me other report (attached) generated with NUnit 2.6.4.14350 that violates the XSD in other part (fix between ).

<xs:complexType name="resultsType">
    <xs:choice **maxOccurs="unbounded"**>
        <xs:element name="test-suite" type="test-suiteType" maxOccurs="unbounded" />
        <xs:element name="test-case" type="test-caseType" maxOccurs="unbounded" minOccurs="0" />
    </xs:choice>
</xs:complexType>

[CharliePoole]

@nfalco79 About this: "I relax the constraint on the original XSD copy on xunit-plugin project but this solve only temporarily because on next release of NUnit this will be broken again."

I don't quite understand the problem. New releases of NUnit don't include the V2 XSD as far as I know. (If they do, they shouldn't) Can you explain further what you mean?

[nfalco79]

Next release of NUnit V2.x

[CharliePoole]

Are you following the nunit-legacy releases? That's good to know. In that case, submit your corrections to the project in an issue! I'd be glad to have them.

[CharliePoole]

TL;DR: If I don't hear back soon, I'll close this issue without fixing.

@nfalco79 You never submitted an issue to the legacy V2 project, so this has been sitting for a long time? Is it still an active problem for you?

FYI I recently took over this project. Although I was commenting here I wasn't responsible for it back in 2018 after some reorganization we did. I noticed this was still active and did some preliminary work. It's a bit of a can of worms, for reasons I'll document here...

1. Back when we created the V2 Result Writer (2008?) we were faced with a quandary. We couldn't precisely map NUnit 3 execution to the NUnit V2 format because the two runners work differently. As a result, we tried to follow the XSD as closely as possible, but relaxed some restrictions. That's why the V2 Result Writer doesn't come with an XSD.

2. In particular, V3 can run multiple projects or a combination of projects and assemblies from the command line. It's top level test-run element (analogous to V2 test-results) may contain many different root files - either projects or assemblies. So we relaxed the requirement to have only a single test-suite element.

3. Although it's quite a bit of work, we could construct a Project element to wrap multiple assemblies but that would be a problem for some V3 command-lines that already contain a project along with other files. That's because NUnit projects are not nestable.

4. Other folks appear to have adapted to the current format. In particular, TeamCity actually creates a project for multiple assemblies on the command-line. I never understood why they did that, but I now believe this issue may actually be the cause!

@NikolayPianikov how would it work for TeamCity if the result writer actually injected a project when multiple assemblies were run from the command-line? I'm imagining it would be OK, as you always run with your own created project.

@nfalco79 This looks somewhat unmanageable for the extension to handle by itself. We could, of course, publish a looser XSD as part of the extension, but that's probably not where Jenkins is looking to find it. I'm also not very clear about whether you are really using the extension or if you are using NUnit V2 itself. If you're using legacy V2, you really have to ask on that project because I try to keep the work there separate from the NUnit Project team, which no longer maintains V2.

On a lighter note, I should mention that this issue, if nothing else, has helped me a lot in improving the tests around the extension! :smile:

[CharliePoole]

I'm modifying the tests to reflect how we handle multiple assemblies currently. I'm assuming we will make no further changes for now, but I'll wait a bit to give @nfalco79 a chance to respond.

[CharliePoole]

Unless there are any objections (@NikolayPianikov ?) I'll modify the legacy NUnit V2 project to loosen the XSD restriction to one top level test suite.

[nfalco79]

I extract XSD from the archives posted on www.nunit.org o maybe sometimes from the source repository (I do not remember). The XSD is saved into the plugin file and there I already patch to allow multple test-suite element under the test-results. This is an example for a test case: https://github.com/jenkinsci/xunit-plugin/blob/master/src/test/resources/org/jenkinsci/plugins/xunit/types/nunit/testcase16/input.xml

[CharliePoole]

@nfalco Earlier, you mentioned "new releases of nunit" so I assumed you were following the nunit legcy project, which is the only place where nunit v2 is being updated.

For the 2.7.2 release at https://github.com/nunit-legacy/nunitv2 I will add maxOccurs="unbounded"to the element in question, but nothing will change on nunit.org. Would you like to see a copy of the xsd somewhere in this project as well?

[CharliePoole]

@ChrisMaddock @nfalco79 @NikolayPianikov @rprouse

I have given this some more thought and I think I changed my mind about the XSD. Here's what I'm now thinking. I'd appreciate your feedback to ensure that nobody is surprised by the change.

1. NUnit V2 always produces output in conformity with the existing XSD. Therefore, it doesn't make sense to change the XSD on the V2 project. I'm thinking I should leave it as is.

2. As noted in this issue, the result writer produces output that is slightly different from the V2 XSD. Therefore, this project should document what it does and make sure users are aware of it. That would include

   • Placing a modified copy of the XSD in the distribution
   • Updating the README to indicate the location of the proper XSD to use
   • Adding a test validate the output of the extension against the XSD

I started to add "Update the docs" to the above list, but I then realized there are no docs for our extensions. That is, the docs tell how to create an extension but I can't find anything about the existing extensions, which we publish. I'll add a separate issue for this.

Even with documentation, I suspect that some users may continue use the wrong XSD and to be surprised at the result, but I don't think we can do anything other than pointing them to the right place when that happens.

[rprouse]

To me, the XSD matches the NUnit 2 runner where it was taken from. As @nfalco79 points out, other runners produce slightly different output. The NUnit 3 runner is yet another runner that is not compliant. To me, the solution would be to support the NUnit 3 result format rather than try to produce NUnit 2 results using NUnit 3.

[CharliePoole]

@rprouse FWIW my V4 recommendation to @ChrisMaddock is that we no longer support the V2 result writer.

Here, I'm looking for an immediate solution in order to wrap this project up in a reasonably complete state for eventual archiving. Then someone else can always elect to fork it and continue.

[NikolayPianikov]

@CharliePoole We are using NUnit projects to make the logic around the tests working directory more predictable. Besides that, it is a workaround for the too-long command line, which may be a potential problem when there are many testing assemblies. To manage several sessions for tests running simultaneously, TeamCity uses a listener that takes this possible parallelism into account and creates its own flow-id for each parallel flow within NUnit. Yes, TeamCity has a report analyzer, but users should do additional stuff to configure it, and they prefer not to do anything, using the listener mentioned above, because it is the simplest and the most robust way. I would prefer to not change anything for legacy NUnit V2.

[CharliePoole]

@NikolayPianikov Exploring some options here...

1. If we only make the change for V4, then isn't there an option for you to not use V4 but stick with V3?
2. What if we continue to support NUnit projects, but freeze the extension - i.e. archive and no longer update it?
3. Would you be willing to take over maintenance as an alternative?
4. Could we set up a project of contributed extensions?
5. What if we replace nunit projects in V4 with a more advanced way to combine projects, possibly a JSON format?

[NikolayPianikov]

@CharliePoole Actually, we can workaround the long command-line by using @files, and we have plans to do it, but there is no way to do the same with the working directory. But perhaps something has already changed. As for me: 1 ++ 2 + 3 ? - I will have to do this if we have no alternative 4 ? - What do you mean? 5 + (plus 2)

[CharliePoole]

@NikolayPianikov Ha Ha! It's like a secret code! But I think I understand and it seems we can work something out.

For this issue, the only action is to fix the problem. Basically, not changing the extension but correcting the XSD and publishing it here.

For the next milestone, 3.7, there are only bug fixes and improvements to the CI. I think that will be the last release of this extension for NUnit 3.

I think the big question is whether to support .nunit projects that include multiple runtime targets, including .NET Core. I think we have to support all that, but I'd prefer to use a different format based on JSON. That would leave you with 5 plus 2, as you said. :smile:

We're talking about a breaking 4.0 release fairly soon and have been discussing this in various 4.0 issues, all listed on the 4.0 milestone. You should probably look at some of them.

PS: What I meant about 4 is that we could put some extensions into a contribution project, like many other open source projects maintain for things that are important to some people but not central to the main project. Then we would only curate the overall set of contributions, while others would work on it. It's just an idea.

[CharliePoole]

:tada: This issue has been resolved in version 3.7.0 :tada:

The release is available on:

• GitHub Release
• NuGet Package
• Chocolatey Package