Integration of NBi and Visual Studio Team Services

Victor-Blaga commented 5 years ago

I'm trying to include an NBi test project in a Visual Studio Team Services pipeline. I create a project locally using the bootstrapper, upload it to Git and I get the following output:

2018-08-06T08:56:51.7007856Z ##[section]Starting: Run Test Scenarios
2018-08-06T08:56:51.7017845Z ==============================================================================
2018-08-06T08:56:51.7018173Z Task         : Visual Studio Test
2018-08-06T08:56:51.7018568Z Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. Tests can be distributed on multiple agents using this task (version 2).
2018-08-06T08:56:51.7019055Z Version      : 2.137.9
2018-08-06T08:56:51.7019246Z Author       : Microsoft Corporation
2018-08-06T08:56:51.7019523Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=835764)
2018-08-06T08:56:51.7019973Z ==============================================================================
2018-08-06T08:56:52.2997017Z Run the tests locally using vstest.console.exe
2018-08-06T08:56:52.2997627Z ========================================================
2018-08-06T08:56:52.2998662Z Test selector : Test assemblies
2018-08-06T08:56:52.2999437Z Test assemblies : **\bin\Debug\test-af.dll,!**\obj\**
2018-08-06T08:56:52.3000001Z Test filter criteria : null
2018-08-06T08:56:52.3000812Z Search folder : D:\a\r1\a
2018-08-06T08:56:52.3001787Z Run settings file : D:\a\r1\a
2018-08-06T08:56:52.3002549Z Run in parallel : false
2018-08-06T08:56:52.3003039Z Run in isolation : false
2018-08-06T08:56:52.3007349Z Path to custom adapters : D:\a\r1\a\drop\test-af\packages\NUnitTestAdapter.2.1.1\tools
2018-08-06T08:56:52.3007902Z Other console options : null
2018-08-06T08:56:52.3008390Z Code coverage enabled : false
2018-08-06T08:56:52.3009487Z Rerun failed tests: false
2018-08-06T08:56:52.3018954Z VisualStudio version selected for test execution : 15.0
2018-08-06T08:56:53.6072737Z ========================================================
2018-08-06T08:56:53.7972517Z [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" @D:\a\_temp\af2b4631-91d6-2fe8-9598-299586afd6da.txt
2018-08-06T08:56:57.4744237Z Microsoft (R) Test Execution Command Line Tool Version 15.7.2
2018-08-06T08:56:57.4748759Z Copyright (c) Microsoft Corporation.  All rights reserved.
2018-08-06T08:56:57.4748982Z 
2018-08-06T08:56:57.5172871Z vstest.console.exe 
2018-08-06T08:56:57.5173419Z "D:\a\r1\a\drop\test-af\test-af\bin\Debug\test-af.dll"
2018-08-06T08:56:57.5173788Z /logger:"trx"
2018-08-06T08:56:57.5174171Z /TestAdapterPath:"D:\a\r1\a\drop\test-af\packages\NUnitTestAdapter.2.1.1\tools"
2018-08-06T08:57:01.2312445Z Starting test execution, please wait...
2018-08-06T08:57:02.4373279Z NUnit VS Adapter 2.1.1.0 executing tests is started
2018-08-06T08:57:03.4175651Z Loading tests from D:\a\r1\a\drop\test-af\test-af\bin\Debug\test-af.dll
2018-08-06T08:57:03.4336594Z Run started: D:\a\r1\a\drop\test-af\test-af\bin\Debug\test-af.dll
2018-08-06T08:57:03.4938185Z NUnit VS Adapter 2.1.1.0 executing tests is finished
2018-08-06T08:57:04.5334776Z No test is available in D:\a\r1\a\drop\test-af\test-af\bin\Debug\test-af.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
2018-08-06T08:57:04.5431193Z 
2018-08-06T08:57:04.5716479Z ##[warning]No results found to publish.
2018-08-06T08:57:04.5827598Z ##[section]Finishing: Run Test Scenarios

I get the same issue locally, if instead of starting the project, I go to the Test Explorer and hit "Run All". There are also no individual tests being displayed in the Test Explorer. This is the output locally:

[2018-08-06 16:14:20 Informational] ------ Discover test started ------
[2018-08-06 16:14:21 Informational] NUnit VS Adapter 2.1.1.0 discovering tests is started
[2018-08-06 16:14:22 Informational] NUnit VS Adapter 2.1.1.0 discovering test is finished
[2018-08-06 16:14:22 Informational] ========== Discover test finished: 0 found (0:00:02.0767881) ==========
[2018-08-06 16:14:24 Informational] ------ Run test started ------
[2018-08-06 16:14:25 Informational] NUnit VS Adapter 2.1.1.0 executing tests is started
[2018-08-06 16:14:25 Informational] Loading tests from <path>\bin\Debug\test-af.dll
[2018-08-06 16:14:26 Informational] Run started: <path>\bin\Debug\test-af.dll
[2018-08-06 16:14:26 Informational] NUnit VS Adapter 2.1.1.0 executing tests is finished
[2018-08-06 16:14:26 Warning] No test is available in <path>\bin\Debug\test-af.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[2018-08-06 16:14:26 Informational] ========== Run test finished: 0 run (0:00:01.3499412) ==========

Is there any way I can get NBi to work in with this setup? Thanks!

Seddryck commented 5 years ago

The dll to test is always NBi.NUnit.Runtime.dll and you need a config file to redirect to your test-suite. In your logs, your setup is clearly trying to load tests from another dll. Not surprised that no tests are found.

I never experimented the compatibility of NBi and Visual Studio Team Services and neither with a VS test adapter. I think a few people have succeed but can’t say more about this. But I successfully integrated NBi and Team City many times.

loewensteinph commented 5 years ago

I am using a Wrapper Assembly to run tests in VS Test see https://github.com/loewensteinph/nBITestRunner

Victor-Blaga commented 5 years ago

Works like a charm using NBi.NUnit.Runtime.dll, I'll give the wrapper assembly a try as well. Thanks, guys! :)

Seddryck commented 5 years ago

@Loewensteinph and @Victor-Blaga if you’ve any additional info on how to successfully integrate NBi and Visual Studio Team Services, feel free to post links, I will add a page to the documentation about integration with CI and will reference your projects/blog posts or anything else.

Victor-Blaga commented 5 years ago

I don't have any blog post, but I can describe the process here. Please bear in mind that I'm no VSTS expert, since I have hardly ever worked with it prior to this project. Also, VSTS is undergoing significant changes, as Microsoft is heavy at work on it.

First of all, I've created a project for Visual Studio 2017 locally, and followed the steps you described in the bootstrapper documentation page. Visual Studio has the option of creating a new Git repository for the project as well, so I made sure that was selected. The Working Directory and Start External Program directories did not map properly from the start, but I reconfigured them using absolute paths.

I copied the solution directory to a path within my Git project's hierarchy, and added it to a testing branch. Whenever there's a new push in this testing branch, the CI is set to redeploy and run tests.

In the Build section of the CI pipeline, I added a NuGet restore step, pointing to the .sln solution file of the testing project. This should make sure you get the same NuGet packages on the VSTS target as you have installed on the local machine.

Next I have created a Visual Studio Build step for the testing project, and pointed it to the same .sln solution file as above. I also checked the "Clean" option, to make sure nothing is left in the output directory from previous builds.

Thirdly, I created a Copy Files step in the Build section of the pipeline, copying everything from $(Build.SourcesDirectory)\\ to $(Build.ArtifactStagingDirectory)\\ in order to have the binaries produced by the build available for deployment.

The final Build step was to create a Publish artifact which contained the binaries above (along with all the SQL deployment scripts and everything else related to the new DW version), which basically moves the files from $(Build.ArtifactStagingDirectory) to some internal VSTS publish location.

The last step is part of the Release section of the VSTS pipeline. I added a Visual Studio Test step there, pointing to **\\bin\Debug\NBi.NUnit.Runtime.dll as a test assembly (thanks for pointing it out). Within the same step I mentioned $(System.DefaultWorkingDirectory) as the Search folder and $(System.DefaultWorkingDirectory)/drop//packages/NUnitTestAdapter.2.1.1/tools as a custom test adapter. The test adapter part is optional, but I think is good to have, since VSTS supports NUnit 3 as well, and might decide to use that as default for whatever reason in the future.

I've attached some screenshots from VSTS for support. I hope this helps with replicating the setup, but please ask if I can help further or have been unclear in any way.

Cheers!

Seddryck commented 5 years ago

Thx for sharing with us.

nobodyForYou commented 5 years ago

Hi Victor-Blaga, I followed your instruction but I have the same message than you with "NBi.NUnit.Runtime.dll":

2018-08-14T20:32:33.1023436Z ====================================================== 2018-08-14T20:32:33.2129140Z [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\SQL\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" @C:\agent\agent_work_temp\2c1a9071-a001-11e8-a69d-65bde9a9b9b4.txt 2018-08-14T20:32:33.6997355Z Microsoft (R) Test Execution Command Line Tool Version 15.6.2 2018-08-14T20:32:33.7005495Z Copyright (c) Microsoft Corporation. All rights reserved. 2018-08-14T20:32:33.7006228Z 2018-08-14T20:32:33.7087130Z vstest.console.exe 2018-08-14T20:32:33.7087580Z "C:\agent\agent_work\r1\a\MyFirstProject-CI\drop\NBi.NUnit.Runtime.dll" 2018-08-14T20:32:33.7087780Z /logger:"trx" 2018-08-14T20:32:33.7087997Z /TestAdapterPath:"C:\agent\agent_work\r1\a" 2018-08-14T20:32:34.3334012Z Starting test execution, please wait... 2018-08-14T20:32:35.0145976Z NUnit Adapter 3.10.0.21: Test execution started 2018-08-14T20:32:35.0233875Z Running all tests in C:\agent\agent_work\r1\a\MyFirstProject-CI\drop\NBi.NUnit.Runtime.dll 2018-08-14T20:32:35.3807398Z NUnit couldn't find any tests in C:\agent\agent_work\r1\a\MyFirstProject-CI\drop\NBi.NUnit.Runtime.dll 2018-08-14T20:32:35.3872422Z NUnit Adapter 3.10.0.21: Test execution complete 2018-08-14T20:32:35.7377822Z No test is available in C:\agent\agent_work\r1\a\MyFirstProject-CI\drop\NBi.NUnit.Runtime.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. 2018-08-14T20:32:35.7552326Z 2018-08-14T20:32:35.8328229Z ##[warning]No results found to publish. 2018-08-14T20:32:35.8613414Z ##[section]Finishing: VsTest - testAssemblies

We don't need to add the config file? And Do you see your test result in vsts?

I have the same message when I execute in cmd : vstest.console.exe path\bin\Debug\NBi.NUnit.Runtime.dll /InIsolation /TestCaseFilter:"Priority=1" /Logger:trx /TestAdapterPath:path\bin\Debug Result: No test is available in path\bin\Debug\NBi.NUnit.Runtime.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Thanks in advance for you response

Victor-Blaga commented 5 years ago

Hey,

I think the issue in this case is the NUnit Adapter version. As far as I know, NBi is supported only by NUnit 2.x, and not by 3.x. Could you try to downgrade the NUnit you are using to the latest 2.x and see if it works that way?

Hope this helps :)

Victor-Blaga commented 5 years ago

To also answer the two questions specifically, I can see test results in VSTS. There is no specific output for tests that pass, but the output is quite verbose for tests that fail, which is perfect in my opinion. Just make sure that you create another test to check this that forces failure. Regarding the config file, at first I didn't use one, when I had less than 5 tests, but after that I introduced one to hold the connection details for the DB. The tests ran either way just fine.

nobodyForYou commented 5 years ago

Nice, I changed UnitTestAdaptater version and it works.

Thanks

Seddryck commented 5 years ago

I'll move this to the official documentation with full credits to @Victor-Blaga

fmms commented 5 years ago

Hi @Seddryck , i presented at SQL Saturday about integrating VSTS and nbi. I just ran the nunit commandline application. Then VSTS will read the TestResults.xml and everything works perfectly. You find the whole deck at http://www.sqlsaturday.com/760/Sessions/Details.aspx?sid=74473 .

Seddryck commented 5 years ago

@Fmms, thx for this new confirmation of the compatibility of NBi with VSTS.

Bhuard1 commented 5 years ago

@Victor-Blaga Hi any chance you can explain how you may have specified the testsuite.config (which contains the testsuite to run and connection strring info) in the visio studio test task.

Currently all tests in .nbits files are run, I need to put all connection string info there (in each file) I cannot figure out how to indicate to test runner/test adapter to read the config file

Victor-Blaga commented 5 years ago

Hi @Bhuard1 , sorry for the delay. The path to your TestSuite.config file should be in TestSuite.nunit

`

Are you trying to run the tests from your own computer, or are they part of an Azure DevOps pipeline?

If you are running them on your computer, go to the bin/Debug or bin/Release folder and check that there is a TestSuite.config file in there too. If not, either copy it over, or change the TestSuite.nunit file in your project to say configfile="../../TestSuite.config".
If you are running tests as part of a pipeline, try downloading the artifact and doing the same verification within the artifact's folder structure. If all still seems OK, but when you run the tests they still cannot find the config file or they say the ConnectionString variable is undefined, maybe check where the build server ends up putting the builds. It might not be /bin/Debug, but I'm not quite sure how to go about doing that.

Hope this helps, let us know if you manage to figure it out.

Bhuard1 commented 5 years ago

@Victor-Blaga

Thank you for your explaination for the testsuite.nunit, I had not looked at that. (I am currently using the TFS on-premise latest version) The testsuite.nunit seems correctly configured and the output directory for my build all seem correct. Does the adapter pick up the nunit project automatically?

I did notice that if the testsuite.config was copied into a file NBi.NUnit.Runtime.dll.config (like other VS console projects I am familiar with) it works. The nunit adpater finds the dll and the config.

For now I have simply copied the testsuite.config to NBi.NUnit.Runtime.dll.config as a post build event.

Probably not how it is intended to work but I am unblocked.

Thanks

sqlsavant commented 4 years ago

I have been through this information many times and still cannot get this to work. I must have something configured incorrectly. Here is the error that I get when I try to run it on my machine:

c:\GIT\HNTB.Database\DeliveryDatamart-IntegrationTest\bin\Release>vstest.console.exe NBi.NUnit.Runtime.dll /InIsolation /Logger:trx /TestAdapterPath:c:\GIT\HNTB.Database\DeliveryDatamart-IntegrationTest\packages Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1 Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait... Information: NUnit VS Adapter 2.2.0.0 executing tests is started

Error: Exception System.IO.FileLoadException, Exception thrown executing tests in c:\GIT\HNTB.Database\DeliveryDatamart-IntegrationTest\bin\Release\NBi.NUnit.Runtime.dll

Information: NUnit VS Adapter 2.2.0.0 executing tests is finished

Warning: No test is available in NBi.NUnit.Runtime.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

Here is the contents of TestSuite.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="nbi" type="NBi.NUnit.Runtime.NBiSection, NBi.NUnit.Runtime"/>
  </configSections>
  <nbi testSuite="TestSuite.nbits"/>
  <connectionStrings configSource="connections.config"/>
</configuration>

Here is the contents of TestSuite.nunit:

<NUnitProject>
  <Settings activeconfig="Default" processModel="Default" domainUsage="Default" />
  <Config name="Default" binpathtype="Auto" appbase="." configfile="TestSuite.config">
    <assembly path="NBi.NUnit.Runtime.dll" />
  </Config>
</NUnitProject>

@Bhuard1 mentioned that he has a NBi.NUnit.Runtime.dll.config file that is created during the build, but I do not. My TestSuite.config, TestSuite.nbits, and TestSuite.nunit do get copied to the bin however.

Any help would be greatly appreciated!

NeilRoper commented 4 years ago

Hi sqlsavant, I have been working through these examples as well to get NUnit tests working on Visual Studio. Unfortunately no luck so far.

I got the same error as you using the NUnit VS Adapter 2.2.0.0, but the older 2.1.1.0 version used in the examples above seemed not to cause this error.

I am currently working through these links to see if I can resolve the problem and will post back if I do: http://hermit.no/how-to-resolve-cases-of-visual-studio-no-tests-appearing/ http://hermit.no/how-to-control-the-selection-of-test-runner-in-tfsvsts-making-it-work-with-x86x64-selected-targets/

Seddryck commented 4 years ago

I don't comment because I've no experience over the integration of NBi and Azure DevOps , only with the integration of TeamCity and NBi. I've not the same kind of issue related to NUnitTestAdapter because Team City allows you to select the native NUnit runner. Sorry can't help.

On the other hand, I'm currently busy to migrate to NBi to NUnit 3.x. This task is clumsy but it's really ongoing and I've the first results (equal-to is migrated). I hope to finalize this task for the end of the year (or sooner). After the migration to NUnit 3.x, I'll take a look to the integration of NBi and Azure DevOps.

NeilRoper commented 4 years ago

@Seddryck That's good news re NUnit 3.x as I think this issue seems to be some small dependency issue between the adapter and the packages referenced in NBi and maybe newer versions would be better. I'm working through the dependencies now and will post if anything comes up.

NeilRoper commented 4 years ago

Only thing I found is that the NUnitTestAdapter.2.2.0 uses version 2.6.4 of nunit.core.dll and nunit.core.interfaces.dll - check: ...\packages\NUnitTestAdapter.2.2.0\build\nunit.core.dll <..2.6.4 ...\packages\NUnitTestAdapter.2.2.0\build\nunit.core.interfaces.dll <..2.6.4

Package NBi.VisualStudio uses version 2.7.0 of these libraries so running the test project (F5) causes a library not found error.

I solved this temporarily by copying the 2.7.0 versions of both files from ...\packages\NUnitV2.Core.2.7.0\lib\ to ...\packages\NUnitTestAdapter.2.2.0\build\

so that each time the projects is built the adapter won't break. Hope this small part helps. I was unable to get the tests running locally on my machine with Visual Studio 2017 however (nothing to do with running this on Azure DevOps).

fmms commented 11 months ago

as @Seddryck did fix a bug that was significant for me with the 1.24 release just this week. Let me share here all my learnings and my configuration.

my test suite NBiTests.nbits is the following:

<?xml version="1.0" encoding="UTF-8"?>
<testSuite xmlns="http://NBi/TestSuite" name="NBi Test Suite">
  <settings>
    <reference name="DWH">
      <connection-string><environment name="DWHConnectionString"/></connection-string>
    </reference>
  </settings>
  <test name="Validate Equality">
    <system-under-test>
      <result-set>
        <query connection-string="@DWH">SELECT 42 AS n</query>
      </result-set>
    </system-under-test>
    <assert>
      <equal-to>
        <result-set>
          <row>
            <cell>42</cell>
          </row>
        </result-set>
      </equal-to>
    </assert>
  </test>
</testSuite>

The Nunit project definition NBiTests.nunit is

<NUnitProject>
  <Settings activeconfig="Default" processModel="Default" domainUsage="Default" />
  <Config
    name="Default"
    binpathtype="Auto" appbase="."
    configfile="NBiTests.config">
    <assembly path="Framework\NBi.NUnit.Runtime.dll" />
  </Config>
</NUnitProject>

The config file NBiTests.config is:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
 <configSections>
  <section name="nbi" type="NBi.NUnit.Runtime.NBiSection, NBi.NUnit.Runtime"/>
 </configSections>
 <nbi testSuite="NBiTests.nbits"/>
</configuration>

My Azure DevOps Pipeline is the following:

                  - task: PowerShell@2
                    displayName: "Download and Install NUnit 2.x"
                    inputs:
                      targetType: 'inline'
                      script: |
                        if (Test-Path 'C:\Program Files (x86)\NUnit 2.7.1\bin\nunit-console.exe') {
                          Write-Host "Nunit is already installed, skipping."
                        } else {
                          $ProgressPreference = 'SilentlyContinue'
                          [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::TLS12
                          $amo = "https://github.com/nunit-legacy/nunitv2/releases/download/2.7.1/NUnit-2.7.1.msi"
                          $amopath = Join-Path $env:TEMP "NUnit-2.7.1.msi"
                          $log = Join-Path $env:TEMP "log.log"
                          Invoke-WebRequest -Uri $amo -OutFile $amopath
                          Write-Output "installing $amopath"
                          Start-Process $amopath -ArgumentList "/q /li $log" -Wait
                          Get-Content $log
                        }

                  - task: PowerShell@2
                    displayName: "Download NBi.Framework"
                    inputs:
                      targetType: 'inline'
                      script: |
                        $ProgressPreference = 'SilentlyContinue'
                        [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::TLS12
                        $amo = "https://github.com/Seddryck/NBi/releases/download/v1.24/NBi-Framework-1.24.0.zip"
                        $amopath = Join-Path $env:TEMP "NBi.Framework.zip"
                        Invoke-WebRequest -Uri $amo -OutFile $amopath
                        Expand-Archive $amopath -Force -DestinationPath "$(Agent.BuildDirectory)\mybuild\arti1\tests\Framework"

                  - ${{ if eq( variables.SSDTTargetEnvironment, 'AzureSQL') }}:
                    - task: AzurePowerShell@5
                      displayName: 'Get Access Token (Azure)'
                      inputs:
                        azureSubscription: ${{ variables.azureserviceconnectionName }}
                        azurePowerShellVersion: LatestVersion
                        ScriptType: 'InlineScript'
                        Inline: |
                          $ato = $(Get-AzAccessToken -Resource "https://database.windows.net/").Token;
                          Write-Host "##vso[task.setvariable variable=ato]$ato"

                  - task: PowerShell@2
                    displayName: 'Run NBi test suite'
                    inputs:
                      targetType: 'inline'
                      script: |
                        $nunit = "$(Agent.BuildDirectory)\mybuild\arti1\tests\NBiTests.nunit"
                        & "C:\Program Files (x86)\NUnit 2.7.1\bin\nunit-console.exe" $nunit
                    env:
                      ${{ if eq(variables.SSDTTargetEnvironment, 'AzureSQL') }}:
                        DWHConnectionString: "Data Source=$(sqlServerFQDN);Initial Catalog=$(databaseName);Provider=MSOLEDBSQL;Access Token=$(ato);"
                      ${{ if eq(variables.SSDTTargetEnvironment, 'onPrem') }}:
                        DWHConnectionString: "Data Source=$(sqlServerFQDN);Initial Catalog=$(databaseName);Provider=MSOLEDBSQL;Integrated Security=SSPI;Auto Translate=False;"

                  - task: PublishTestResults@2
                    displayName: "Publish NBi tests"
                    condition: always()
                    inputs:
                      testResultsFormat: 'NUnit'
                      testResultsFiles: '**/TestResult.xml'
                      failTaskOnFailedTests: true

Looking forward for any feedback, for me this seems like a very nice Azure DevOps integration.

One can even nicely see the reason for failed test cases:

Seddryck / NBi

Integration of NBi and Visual Studio Team Services #399