nunit / nunit3-vs-adapter

NUnit 3.0 Visual Studio test adapter for use under VS 2012 or later
https://nunit.org
MIT License
202 stars 104 forks source link

Failed to load extensions from file NUnit3.TestAdapter.dll error in Azure DevOps Release pipeline ONLY #1151

Closed johnjoncyn closed 3 months ago

johnjoncyn commented 5 months ago

I have created a vs 2022 project. The tests run sucessfully in VS2022 and Azure DevOps Build Pipeline.

The same fails when i run it in the release pipeline with the output of a sucessful build.

Here are the details Microsoft Visual Studio Professional 2022 (64-bit) - Current Version 17.8.6 Project

net8.0
OsirisTerje commented 5 months ago

There is no extensions to the NUnit3Testadapter, so it doesn't make sense that it asks for that.

Notices that you reference the Microsoft.TestPlatform.TestHost, that should not be included.

Can you upload the yml file you're using ? And the csproj for your project.

If you're using the old classic build you should convert to yml.

I see a message there refer to "new test platform". If that is the completely new test platform that was announced a few days ago, that should be turned off.

johnjoncyn commented 5 months ago

@OsirisTerje Thanks for the response. I have attached the yml and proj file for the build pipline where the automated Test Cases run sucessfully with no errors. files.zip

Where would i find the details of config for the test platform.

In the release pipeline it dosent work as expected and i havent used yml to configure the release pipeline. please see the screenshot in the above description of what the release pipeline looks like.

OsirisTerje commented 5 months ago

The yml and csproj looks good.

The Azure DevOps release pipeline doesn't use YML files (afaik) , and is based off the old classic model. I am not sure what exactly it uses these days, and I don't think it is much in use for new systems today. It might use the old tests tasks using VSTest.console, or some derivative of that. I am not sure at all here.

The way people do today is to add releases into the build pipeline, using yml files to describe them. That would work for you too as the current build pipeline works. There are a lot of information available on how to deploy using the build pipeline.

If you want to debug the release pipeline further, you need to come up with more information on what exactly goes wrong. Which task, and what is its configuration. With the classic model, it would be a ton of screenshots. As said above, the adapter doesnt have extensions, so it is not likely to create the crash itself. It must be some other component trying to do something it should not even try to do.

I would also consider why you would run tests in the release pipeline at all. They belong more in a build pipeline. The release is about deploying stuff somewhere.

johnjoncyn commented 5 months ago

@OsirisTerje That right. I am using the classic version. The objective is to use the release pipeline to update the Test Plan module in Azure devops. Test run in the test plan cant be updated with the test results from the Build.

Its very basic setup using an agent configured to run on my windows 10 and your right i an using visual studio test. The screenshot above shows the config. whats suprising is that it works in the build pipeline but not release.

OsirisTerje commented 5 months ago

Your build pipeline is running dotnet test and not vstest.console. I assume that the release pipeline is using the vstest.console, and there might be a difference there.

You can try locally to run either dotnet vstest or even vstest.console and see if that repro the crash.

I can't see why the build pipeline should not be able to update the Test Plan module. It has an API like everything else there.

UPDATE:

If you look into https://github.com/Microsoft/vstest/blob/main/src/Microsoft.TestPlatform.VsTestConsole.TranslationLayer/VsTestConsoleWrapper.cs you can see this is the test console wrapper, and it has an extension mechanism (see line 49). It is not the new system, as I suspected earlier. So this is what is crashing. It might be wise to create an issue in that repo.

It should also be possible to repro this locally as I mentioned above.

You can also try to drop down to .net 6 and an earlier version of the SDK to see if that makes any difference here.