ITestDataSource is being called unnecessarily during test discovery, and when a single test is executed

[ttorkelson]

Description

Since version 2.2.3, ITestDataSource is being called unnecessarily during test discovery, and when a single test is executed. My organization creates/retrieves many resources within the GetData() call and they should only be executed when they are expected to be.

Steps to reproduce

Create a blank mstestv2 project and paste the following into a single file. you must pivot from mstest version 2.2.3 -> 2.2.8 to demonstrate the failure. There are three tests, all decorated with the same ITestDataSource.

Version	Scenario	dotnet test command	Expected ITestDataSource:TestMethod calls	Actual
2.2.3	Run all tests	dotnet test	3:3	3:3
2.2.3	Run only first test	dotnet test --filter Name~First	1:1	1:1
2.2.3	list tests	dotnet test --list-tests	0:0	0:0
2.2.4+	Run all tests	dotnet test	3:3	3:3
2.2.4+	Run only first test	dotnet test --filter Name~First	1:1	3:1
2.2.4+	list tests	dotnet test --list-test	0:0	3:0

Test code for reproduction

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Reflection;

namespace demo
{
    public class TestData : Attribute, ITestDataSource
    {
        private static readonly object Lock = new();

        public IEnumerable<object[]> GetData(MethodInfo methodInfo)
        {
            lock (Lock)
            {
                Trace.Write(methodInfo.Name);
                System.Threading.Thread.Sleep(5000);
            }
            return new[] {
                new object[] { "foo" },
            };
        }

        public string GetDisplayName(MethodInfo methodInfo, object[] data)
        {
            return $"{methodInfo.Name} ({string.Join(',', data)})";
        }
    }

    [TestClass]
    public class DataSourceRatioTests
    {
        public TestContext TestContext  { get; set; }

        [AssemblyCleanup]
        public static void AssemblyCleanup() {
            Trace.Delete();
        }

        [TestMethod]
        [TestData]
        public void FirstTest(string data)
        {
            Assert.AreEqual("FirstTest", Trace.Read(), "Unexpected attribute calls for first test");
        }

        [TestMethod]
        [TestData]
        public void SecondTest(string data)
        {
            Assert.AreEqual("FirstTest,SecondTest", Trace.Read(), "Unexpected attribute calls for second test");
        }

        [TestMethod]
        [TestData]
        public void ThirdTest(string data)
        {
            Assert.AreEqual("FirstTest,SecondTest,ThirdTest", Trace.Read(), "Unexpected attribute calls for third test");
        }
    }

    public static class Trace
    {
        public static void Delete()
        {
            File.Delete("output");
        }

        public static void Write(string message)
        {
            File.AppendAllLines("output", new string[] { message });
        }

        public static string Read()
        {
            return string.Join(',', File.ReadAllLines("output"));
        }
    }
}

Environment

dotnet --info .NET SDK (reflecting any global.json): Version: 6.0.103 Commit: 2d7bc7059f

Runtime Environment: OS Name: Windows OS Version: 10.0.19044 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\6.0.103\

Host (useful for support): Version: 6.0.3 Commit: c24d9a9c91

.NET SDKs installed: 3.1.404 [C:\Program Files\dotnet\sdk] 3.1.417 [C:\Program Files\dotnet\sdk] 5.0.104 [C:\Program Files\dotnet\sdk] 5.0.212 [C:\Program Files\dotnet\sdk] 5.0.404 [C:\Program Files\dotnet\sdk] 5.0.405 [C:\Program Files\dotnet\sdk] 5.0.406 [C:\Program Files\dotnet\sdk] 6.0.103 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.23 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[Haplois]

Hi,

This is expected since we expand data tests during discovery. For us to be able to discover all tests we need to read the data too.

You can opt-out of this behavior as described https://github.com/microsoft/testfx/issues/905#issuecomment-907775327 by adding following to your solution:

[assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)]

[ttorkelson]

@Haplois Thanks for the workaround, I didn't even know it existed!