search

microsoft / testfx

MSTest framework and adapter
MIT License
679 stars 246 forks source link

3.4 is not working with Fakes #3003

Open Evangelink opened 1 month ago

Evangelink commented 1 month ago

3.4.0 and 3.4.1 are both not working while 3.3.1 was fine for this scenario:

The Fakes assemblies cannot be resolved, each test that uses them outputs:

System.IO.FileNotFoundException: Could not load file or assembly 'ProjectX.Fakes, Version=x.x.x.x, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified..

Where ProjectX.Fakes.dll is present in the output folder next to ProjectY.exe.

Originally posted by @Rans4ckeR in https://github.com/microsoft/testfx/issues/2922#issuecomment-2134633254

Evangelink commented 1 month ago

Could you please provide some more details? Are you using VSTest or MSTest runner to run the tests? Is the issue happening only on VS? Would it be possible for you to create a small reproducer?

cc @drognanar @jakubch1

Rans4ckeR commented 1 month ago

Could you please provide some more details? Are you using VSTest or MSTest runner to run the tests? Is the issue happening only on VS? Would it be possible for you to create a small reproducer?

cc @drognanar @jakubch1

I can only reproduce it on Azure DevOps, not locally;

Pipeline bits

pool:
  vmImage: 'windows-2022'

variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'
  Agent.Source.Git.ShallowFetchDepth: 0

steps:
- task: UseDotNet@2
  inputs:
    packageType: 'sdk'
    version: '8.x'

- task: VSBuild@1
  displayName: 'Build Solution'
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    maximumCpuCount: true
    msbuildArchitecture: 'x64'

- task: CmdLine@2
  displayName: 'Run tests'
  inputs:
    script: '.\TestProject\bin\Release\net8.0\TestProject.exe --report-trx --coverage'

ProjectX

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <GenerateDocumentationFile>true</GenerateDocumentationFile>
  </PropertyGroup>
  <ItemGroup>
    <InternalsVisibleTo Include="TestProject" />
    <InternalsVisibleTo Include="ProjectX.Fakes" />
  </ItemGroup>
</Project>

TestProject

<Project Sdk="MSTest.Sdk/3.4.1">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <GenerateDocumentationFile>true</GenerateDocumentationFile>
  </PropertyGroup>
  <ItemGroup>
    <Compile Remove="FakesAssemblies\**" />
    <EmbeddedResource Remove="FakesAssemblies\**" />
    <None Remove="FakesAssemblies\**" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.QualityTools.Testing.Fakes" Version="17.5.0-beta.23060.1" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\ProjectX\ProjectX.csproj" />
  </ItemGroup>
  <ItemGroup>
    <AdditionalDesignTimeBuildInput Remove="FakesAssemblies\**" />
  </ItemGroup>
</Project>

TestProject\Fakes\ProjectX.fakes

<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/">
  <Assembly Name="ProjectX"/>
  <StubGeneration>
    <Clear/>
    <Add Interfaces="true" Namespace="ProjectX!"/>
  </StubGeneration>
  <ShimGeneration Disable="true"/>
</Fakes>

If I build & run the tests locally in VS it works. Both running the tests in VS and running the .exe directly works. If I copy the binaries produced from the pipeline to my local machine I get the same assembly resolution error.

Also tried this in the pipeline with the same result:

drognanar commented 1 month ago

what's the VS version on your local machine vs your CI? we recently updated the MSBuild props/targets files to include the Fakes dependencies in the executable's runtime configuration

with older VS versions, assembly resolution only works via vstest.console.exe, because it has a custom assembly resolver and picks up the Fakes assemblies regardless of the runtime configuration file content.

Rans4ckeR commented 1 month ago

what's the VS version on your local machine vs your CI? we recently updated the MSBuild props/targets files to include the Fakes dependencies in the executable's runtime configuration

with older VS versions, assembly resolution only works via vstest.console.exe, because it has a custom assembly resolver and picks up the Fakes assemblies regardless of the runtime configuration file content.

On local machine 1 I have Microsoft Visual Studio Enterprise 2022 (64-bit) - Current Version 17.10.0 On local machine 2 I have Microsoft Visual Studio Enterprise 2022 (64-bit) - Preview Version 17.11.0 Preview 1.0 The CI machine, at the time of writing, has Visual Studio Enterprise 2022 | 17.9.34902.65

So it is expected that Fakes with 3.4.x does not work with VS <17.10 unless using VSTest@2?

drognanar commented 1 month ago

Yes, that's expected.

You'd need to have a custom assembly resolver to resolve the dlls from .exe's folder without the correct runtime configuration.

Rans4ckeR commented 1 month ago

Yes, that's expected.

You'd need to have a custom assembly resolver to resolve the dlls from .exe's folder without the correct runtime configuration.

So it's an undocumented breaking change, since 3.3.1 works fine?

drognanar commented 1 month ago

if it's been working via a .exe already then there must be a change in how MSTest itself resolves assemblies

@evangelink are you aware of any changes to assembly resolution in MSTest?

Evangelink commented 1 month ago

Outside of the bugs of 3.4.0 where the resolver wasn't triggered or the infinite recursion with resources not that I am aware of.

cbersch commented 1 month ago

Outside of the bugs of 3.4.0 where the resolver wasn't triggered or the infinite recursion with resources not that I am aware of.

@Evangelink What infinite recursion with resources are you referring to? I couldn't find any related bug. We are expiriencing a problem on Azure DevOps with

The active test run was aborted. Reason: Test host process crashed : Process terminated. Encountered infinite recursion while looking up resource 'Arg_NullReferenceException' in System.Private.CoreLib. Verify the installation of .NET is complete and does not need repairing, and that the state of the process has not become corrupted.

Both with MSTest 3.3.1 and 3.4.0. However I cannot directly relate it to MSTest

Evangelink commented 1 month ago

@Evangelink What infinite recursion with resources are you referring to? I couldn't find any related bug.

I am referring to #2692.

The active test run was aborted. Reason: Test host process crashed : Process terminated. Encountered infinite recursion while looking up resource 'Arg_NullReferenceException' in System.Private.CoreLib. Verify the installation of .NET is complete and does not need repairing, and that the state of the process has not become corrupted.

Could you please provide either a repro or diagnostic logs (see https://github.com/microsoft/vstest/blob/main/docs/diagnose.md)

cbersch commented 1 month ago

@Evangelink What infinite recursion with resources are you referring to? I couldn't find any related bug.

I am referring to #2692.

Ok, that is a different one.

The active test run was aborted. Reason: Test host process crashed : Process terminated. Encountered infinite recursion while looking up resource 'Arg_NullReferenceException' in System.Private.CoreLib. Verify the installation of .NET is complete and does not need repairing, and that the state of the process has not become corrupted.

Could you please provide either a repro or diagnostic logs (see https://github.com/microsoft/vstest/blob/main/docs/diagnose.md)

If I can reproduce it reliably, I'll create an issue. Thanks.

Evangelink commented 1 month ago

Awesome, thank you @cbersch

Rans4ckeR commented 1 month ago

So it is expected that Fakes with 3.4.x does not work with VS <17.10 unless using VSTest@2?

Yes, that's expected.

You'd need to have a custom assembly resolver to resolve the dlls from .exe's folder without the correct runtime configuration.

VS was updated to 17.10 by https://github.com/actions/runner-images/issues/9965 but I still see an 'invalid' .exe being generated with 3.4.1 which can't resolve the Fakes assemblies.

if it's been working via a .exe already then there must be a change in how MSTest itself resolves assemblies

@Evangelink are you aware of any changes to assembly resolution in MSTest?

The .exe generated by 3.3.1 works fine on Azure DevOps with VS 17.10.

Evangelink commented 3 weeks ago

ping @drognanar

drognanar commented 2 weeks ago

I just pushed a new package https://www.nuget.org/packages/Microsoft.Testing.Extensions.Fakes that should support Fakes and the new MSTest runner working side by side.

If you replace the existing Microsoft.QualityTools.Testing.Fakes package reference with the new extension reference it will pick up all of the environment variables as well as add the relevant instrumentation files to the bin folder to ensure that shims can be executed.

Rans4ckeR commented 2 weeks ago

I just pushed a new package https://www.nuget.org/packages/Microsoft.Testing.Extensions.Fakes that should support Fakes and the new MSTest runner working side by side.

If you replace the existing Microsoft.QualityTools.Testing.Fakes package reference with the new extension reference it will pick up all of the environment variables as well as add the relevant instrumentation files to the bin folder to ensure that shims can be executed.

I think you forgot to push the dependency Microsoft.QualityTools.Testing.Fakes (>= 17.11.0-beta.24318.5)?

drognanar commented 2 weeks ago

Thanks for letting me know. Working on it right now, I need to update the version of the Microsoft.Testing.Platform that the package depends on as well. This requires building new packages.

Will update on the thread, once new packages are built and published.

drognanar commented 2 weeks ago

Updated both of the packages

Rans4ckeR commented 2 weeks ago

Updated both of the packages

The result seems to be the same with MSTest.Sdk/3.4.3 and Microsoft.Testing.Extensions.Fakes 17.11.0-beta.24319.3:

Unable to create instance of class TestClassY. Error: System.IO.FileNotFoundException: Could not load file or assembly 'XXX.Fakes, Version=x.x.x.x, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified..
Stack Trace:
at TestClassY..ctor()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

MSTest.Sdk/3.3.1 and Microsoft.Testing.Extensions.Fakes 17.11.0-beta.24319.3 works fine.

Are you perhaps able to spin up an Azure DevOps build yourself and reproduce it?

drognanar commented 2 weeks ago

I rolled back the targets file change locally and was able to reproduce it.

By bisecting the MSTest repo, the first package where this behaviour changed is this one https://github.com/microsoft/testfx/pull/2654, so I'll let @evangelink take a further look

Evangelink commented 2 weeks ago

@Rans4ckeR could you please confirm that when using a more recent version of VS the extension is working fine?

Rans4ckeR commented 2 weeks ago

@Rans4ckeR could you please confirm that when using a more recent version of VS the extension is working fine?

When locally using VS 17.10 or VS 17.11 everything always works, regardless of the versions of any of the packages involved. Tests are running fine inside VS and tests are running fine when using the compiled .exe.

When compiling on Azure DevOps using VSBuild@1 the compiled .exe has different assembly probing logic than the one compiled locally. Only when reverting to MSTest.Sdk/3.3.1 the Azure DevOps .exe uses correct assembly probing.

The current runners on Azure DevOps are using VS 17.10.

The difference might be that VSBuild@1 is using MSBuild.exe while locally dotnet.exe is always used or something?

Evangelink commented 2 weeks ago

The difference might be that VSBuild@1 is using MSBuild.exe while locally dotnet.exe is always used or something?

I guess the difference is the version of MSBuild being inserted/available in dotnet SDK compared to the one in VS.

For the change in the version of MSTest. We introduced support of assembly resolution for .NET Core in #2110 but received many bugs related to this new feature as it's impacting how assemblies are being resolved by .NET so we decided to only plug the assembly resolver (for .NET Core) when user is asking for AssemblyResolution support in the .runsettings. This means that you ended up in the happy path of fakes working with MSTest 3.3 thanks to the assembly resolver although it wasn't designed for it. On MSTest 3.4+, we have this logic of using the resolver only when needed.

You will either need to wait for AzDO to contain the correct preview version of VS that contains the fix (@drognanar could you please confirm the version) or you will need to use AssemblyResolution functionality (https://learn.microsoft.com/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file#mstest-element) pointing to your bin folder.

drognanar commented 2 weeks ago

The Fakes changes that makes Fakes assemblies load even without a custom assembly loader have been merged in 17.11-preview 1.