DeploymentItem doesn't work for pure test class without test methods

N-Olbert commented 5 years ago

Description

According to the documentation the DeploymentItem can

be specified on test class or test method (src)

Anyways, when applying the DeploymentItem to a TestClass which only contains a (valid) AssemblyInitialize- or AssemblyCleanup-Method, the specified resource won't be deployed.

This seems due to the fact that currently the deployment item is only evaluated for TestCase-items (see src/Adapter/PlatformServices.Shared/netstandard1.3/Utilities/ns13DeploymentUtilityBase.cs line 52).

Either the documentation should be updated or the deployment item should be evaluated for any class regarding the existence of a test method. Another solution could be enabling the DeploymentItem attribute to target assemblies.

Steps to reproduce

    [DeploymentItem("Resource.bin")]
    [TestClass] //Note the TestClass-attribute
    public class Class1
    {
        [AssemblyInitialize]
        public static void AssemblyInit(TestContext context)
        {
            Assert.IsTrue(File.Exists("Resource.bin")); //Fails
        }
    }

Expected behavior

Assert.IsTrue succeeds

Actual behavior

Assert.IsTrue fails

Workaround

As a workaround an empty bodied test method can be added. Then it works as expected:

    [DeploymentItem("Resource.bin")]
    [TestClass] //Note the TestClass-attribute
    public class Class1
    {
        [AssemblyInitialize]
        public static void AssemblyInit(TestContext context)
        {
            Assert.IsTrue(File.Exists("Resource.bin")); //Fails
        }
        [TestMethod]
        public void ForceDeployment() { }
    }

AB#1681020

vagisha-nidhi commented 5 years ago

Hello @N-Olbert, I have tried the exact steps you mentioned here but I am unable to repro this. Even with just the AssemblyInitilize in the class, it is able to deploy the items. Can you share a sample repro project for us to investigate further?

N-Olbert commented 5 years ago

@vagisha-nidhi sorry for the inconvenience.

I attached a demo project where the beahviour can be reproduced. UnitTestProject1.zip

Anyways this bug seems pretty clear as the deployment operations are done a "per TestCase"-base. So if no test case exists nothing can be deployed for a class, even if it is markes as TestClass or contains an AssemblyInitialize-Attribute.

https://github.com/microsoft/testfx/blob/3b5926f8437a03744a641eb6dfe38e3aa7cd90bc/src/Adapter/PlatformServices.Shared/netstandard1.3/Services/ns13TestDeployment.cs#L114-L125

Regarding the Hotfix of the emtpy method: This even seems to work if you specify the Ignore-Attribute.

N-Olbert commented 5 years ago

One more note: The hotfix only works if you choose "Run all" within the Visual Studio Test Explorer Window. If you run a single unit test ("Run Selected Tests") the hotfix won't work.

Reason for this is the following: If you use "Run all" the following method will be called:

https://github.com/microsoft/testfx/blob/3b5926f8437a03744a641eb6dfe38e3aa7cd90bc/src/Adapter/MSTest.CoreAdapter/Execution/TestExecutionManager.cs#L86 Therefore MSTest Adapter discovers all possible tests from the given sources (also the hotfix test) and performs the deployment -> everything is ok.

If you use "Run Selected Tests" a different method gets called which takes a collection of test cases to execute as parameter. If the hotfix method isn't included in this collection the hotfix cannot be applied.

https://github.com/microsoft/testfx/blob/3b5926f8437a03744a641eb6dfe38e3aa7cd90bc/src/Adapter/MSTest.CoreAdapter/Execution/TestExecutionManager.cs#L63

vagisha-nidhi commented 5 years ago

@N-Olbert We agree this is a bug. But may I know what is the exact scenario you want achieve with this? Is this a blocking issue? We can decide to take up the fix or change the documentation as per the requirements then.

N-Olbert commented 5 years ago

@vagisha-nidhi Basically the idea was to get a low-impact workaround while migrating from MSTestV1 to MSTestV2. In MSTestV1 we had a .testsettings file which used LegacySettings<LegacySettings> <Deployment> <DeploymentItem filename="..."/> </Deployment> </LegacySettings> to deploy some (large and often changing) files that nearly all tests need.

Adding the according DeploymentItems to around 11k classes wasn't a real option, so we created one class which contains all DeploymentItems and an AssemblyInitialize-Method to ensure the deployment.

I think what we really would have needed would be a DeploymentItem targeting assembly scope.

Anyways, we can live with that empty body Hotfix, but it's not very clean and a discrepancy with the documentation which is why I created the issue.

vagisha-nidhi commented 5 years ago

@N-Olbert There is one more thing you can try out. You can keep your files (that you are using to deploy) in your output directory(bin/debug) and use this setting node your runsettings file.

<MSTestV2>
    <DeployTestSourceDependencies>false</DeployTestSourceDependencies>
</MSTestV2>

What this basically does is that it will copy every thing from your output directory into the deployment directory (won't figure out dependencies on its own) and run tests from there. I will log a doc bug to address this issue.

builder-main commented 4 years ago

One more note: The hotfix only works if you choose "Run all" within the Visual Studio Test Explorer Window. If you run a single unit test ("Run Selected Tests") the hotfix won't work.

Reason for this is the following: If you use "Run all" the following method will be called:

https://github.com/microsoft/testfx/blob/3b5926f8437a03744a641eb6dfe38e3aa7cd90bc/src/Adapter/MSTest.CoreAdapter/Execution/TestExecutionManager.cs#L86

Therefore MSTest Adapter discovers all possible tests from the given sources (also the hotfix test) and performs the deployment -> everything is ok. If you use "Run Selected Tests" a different method gets called which takes a collection of test cases to execute as parameter. If the hotfix method isn't included in this collection the hotfix cannot be applied.

https://github.com/microsoft/testfx/blob/3b5926f8437a03744a641eb6dfe38e3aa7cd90bc/src/Adapter/MSTest.CoreAdapter/Execution/TestExecutionManager.cs#L63

We are experiencing the same behaviour. When running all tests our items are correctly deployed to the test directory. However if we run a single test then no deployment occurs at all (the Out directory is empty)

I tried the following in our testsettings but it didn't work

<MSTestV2>
    <DeployTestSourceDependencies>false</DeployTestSourceDependencies>
</MSTestV2>

TheXenocide commented 3 years ago

In case anyone else needs to know, I was hit by this bug recently. The No-Op TestMethod was a suitable enough workaround for our needs. In our case we have a variety of files coming in via props and targets files included in the project via NuGet packages (config files, some generated code, and so on).

For anyone who spends even half as much time searching the world over through people's questions/answers as I just did, I'll note for the index bots (and hopefully time save readers) that:

Our files were successfully copying to the output directory (a very common "make sure you have this set" bit of guidance around the webs).
We don't use a testsettings or runsettings file in any of our other projects
We never needed /noisolation or LegacySettings
We never needed to Enable Deployment for tests to work correctly before
All paths were relative to the output folder of the assembly; it seems like maybe once upon a time they were relative to the solution directory or the test run directory or who knows where else, but at least in modern times the attribute value is relative to the build output directory (e.g. bin\Debug or bin\Debug\x64)

I couldn't figure out why the one project would be behaving differently enough that all guidance anywhere on the web would push us toward using these files (whether by this more modern DeployTestSourceDependencies workaround which finally lead me to this page). In our case the issue was that the test project needed to run in x64 which was forcing deployment to a separate test directory rather than out of the output directory of the project.

Evangelink commented 1 year ago

I confirm that I can reproduce the issue raised here. Given the few upvotes and the fact this is "broken" since a long time, I will move forward by updating the documentation to mention this is not a supported behavior.

microsoft / testfx