Support xUnit 2.0.0 runners in Bau.Xunit

[jasonmitchell]

It appears that the latest console runner for xUnit has changed the format of arguments to expect that they are prefixed with "-" rather than "/"; e.g. "-xml" instead of "/xml". This results in the following error when running the xUnit task:

error: file not found: /xml

I have made some proof of concept changes in my fork of the repository and tested an updated version of Bau.Xunit against the latest console runner. You can find it here.

My concern about this change is that if someone were to update their Bau.Xunit task without updating their xUnit runners it could potentially break the script.

If you are happy with the changes I can submit a pull request.

[adamralph]

@jasonmitchell great stuff! This actually struck me the other day whilst I was talking to someone about Bau :smile:

Backwards compatibility is definitely a concern. I don't think we can just change the package to be v2 only since a lot of people have not yet migrated to xunit 2. There are many reasons for this, e.g. integration packages like AutoFixture.Xunit still only support v1. There is also currently an issue running v2 on Mono. For these reasons I think the package has to go on supporting v1 for some time yet. I wonder what the best way is to handle this. A few options spring to mind:

1. A new XunitVersion property on the task, defaulting to 1
2. Auto detecting the version by examining the exe. This could be done using a reflection only load, for example, and checking the version in the assembly metadata
3. A new package Bau.Xunit2

Option 2 sounds quite interesting since it would be a completely seamless experience for users. I guess it would have to spiked to see if it's feasible.

[aarondandy]

I also would like to see Option 2. Option 3 seems like it may add confusion unless the original package is renamed to include 1 in the name. If option 1 is used I think that a default of v2 would be best as Bau has not yet reached v1 and v1 will probably have a similar lifetime to xunit v2, but that is just my opinion.

[adamralph]

There is also another option:

1. ...
2. ...
3. ...
4. A new task type, i.e. bau.Xunit2(...)

Another thing to consider - how well do the v2 command line options match with v1?

V1:

xUnit.net console test runner (64-bit .NET 4.0.30319.34209)
Copyright (C) 2013 Outercurve Foundation.

usage: xunit.console.clr4 <xunitProjectFile> [options]
usage: xunit.console.clr4 <assemblyFile> [configFile] [options]

Valid options:
  /silent                : do not output running test count
  /teamcity              : forces TeamCity mode (normally auto-detected)
  /wait                  : wait for input after completion
  /trait "name=value"    : only run tests with matching name/value traits
                         : if specified more than once, acts as an OR operation
  /-trait "name=value"   : do not run tests with matching name/value traits
                         : if specified more than once, acts as an AND operation

Valid options for assemblies only:
  /noshadow              : do not shadow copy assemblies
  /xml <filename>        : output results to Xunit-style XML file
  /html <filename>       : output results to HTML file
  /nunit <filename>      : output results to NUnit-style XML file

V2:

xUnit.net console test runner (64-bit .NET 4.0.30319.34209)
Copyright (C) 2015 Outercurve Foundation.

usage: xunit.console <assemblyFile> [configFile] [assemblyFile [configFile]...] [options]

Note: Configuration files must end in .config

Valid options:
  -parallel option       : set parallelization based on option
                         :   none - turn off all parallelization
                         :   collections - only parallelize collections
                         :   assemblies - only parallelize assemblies
                         :   all - parallelize assemblies & collections
  -maxthreads count      : maximum thread count for collection parallelization
                         :   0 - run with unbounded thread count
                         :   >0 - limit task thread pool size to 'count'
  -noshadow              : do not shadow copy assemblies
  -teamcity              : forces TeamCity mode (normally auto-detected)
  -appveyor              : forces AppVeyor CI mode (normally auto-detected)
  -nologo                : do not show the copyright message
  -quiet                 : do not show progress messages
  -wait                  : wait for input after completion
  -debug                 : launch the debugger to debug the tests
  -serialize             : serialize all test cases (for diagnostic purposes only)
  -trait "name=value"    : only run tests with matching name/value traits
                         : if specified more than once, acts as an OR operation
  -notrait "name=value"  : do not run tests with matching name/value traits
                         : if specified more than once, acts as an AND operation
  -method "name"         : run a given test method (should be fully specified;
                         : i.e., 'MyNamespace.MyClass.MyTestMethod')
                         : if specified more than once, acts as an OR operation
  -class "name"          : run all methods in a given test class (should be fully
                         : specified; i.e., 'MyNamespace.MyClass')
                         : if specified more than once, acts as an OR operation
  -xml <filename>        : output results to xUnit.net v2 style XML file
  -xmlv1 <filename>      : output results to xUnit.net v1 style XML file
  -nunit <filename>      : output results to NUnit-style XML file
  -html <filename>       : output results to HTML file

Things are very different, but it does seem that v2 is a superset of v1 with the exception of v1 silent which maps to v2 quiet.

One clear difference is that v2 can be passed a list of assemblies instead of just one. Fortunately the API already allows for a list of assemblies. For v1, we loop through them invoking xunit for each separately but for v2 we can just pass them all at once.

[adamralph]

BTW - I still favour option 2 if it can be proven to work.

[aarondandy]

It looks like with the exception of silent, v2 is a superset so that should work great. My only concern is that we would be pushing this versioning problem down the road to xUnit 3.

Maybe a safe approach is to have version be a configurable option, that when left blank would lead down the discovery path. bau.Xunit(...).Version1()... .

If in xUnit 3 silent is removed, we could detect that the runner is v3 (if not explicitly set by the user) and throw a NotSupportedException or something. If silent is kept but the semantics/values are changed then I guess that could get messy.

[jasonmitchell]

From my perspective (as someone new to Bau) I would prefer option 2 if it's feasible. The less messing about configuring versions the better if you ask me. I like the safety and flexibility provided by the optional configuration option with a fallback to auto-discovery @aarondandy mentioned above.

[adamralph]

It's a good point about the semantics of silent and quiet. Even though they have the same outcome in v1 and v2 we should respect the semantics.

So, I propose:

• throw when quiet is used with v1 or silent is used with v2
• throw when any other v2-only option is used with v1
• the exception type should be NotSupportedException and the message should say something like "The '{0}' option is not supported by '{1}' version {2}.", option, exe, version

At the moment, I don't see any reason for way to control the version via the API. If auto-detection works then let's handle this internally. If there are any problems with auto-detection then we can add something to the API.

[aarondandy]

@jasonmitchell @adamralph I think we are all mostly in agreement on this, at least enough for the issue to be worked.