TestCase DataSource provider is not available for MStestv2

[sbaid]

Description

DataSource provider - "Microsoft.VisualStudio.TestTools.DataSource.TestCase" is not available for mstestv2. Need a custom data source provider for TFS TestCase datasource. [DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase",             "https://.visualstudio.com/DefaultCollection;",             "",             DataAccessMethod.Sequential)]

Steps to reproduce

Refer below links about documentation on how to use TestCase DataSource with mstestv1 & CodedUI test. While Coded UI tests should continue to reference MSTestv1 and cannot be upgraded to use MStestv2, the documentation gives repro steps how to use TestCase datasource provider with tests other than CodedUI tests. Such tests can use the data source to fetch test parameters from TFS test case.

https://msdn.microsoft.com/en-us/library/ee624082.aspx https://social.msdn.microsoft.com/Forums/en-US/f21b9a8a-87b1-4f9f-b147-6839c7517f51/how-to-parameterize-codedui-tests-from-test-manager?forum=vsautotest how to use the datasourceprovider.

[AbhitejJohn]

I think we should route this through the DataSource extensibility we added in. Docs here. Tagging @harshjain2, @jayaranigarg for anyone who needs help getting this up.

[l3oferreira]

This may help https://github.com/l3oferreira/MSTest.AzureDevOps

[koskit]

Can we have some attention to this problem? We want to upgrade to MSTest v2, but this issue is a blocker.

We have to configure more than 800+ test cases to accept a parameter if we are to extend DataRowAttribute or use ITestDataSource AND pass it to our BaseTest class that distributes the data on multiple classes.

The advantage we have right now, is CodedUITests with DataSource from TFS automatically handle iterations from the count of DataRows, and each row is individually set to the context for the test to take data.

Can we have an implementation for extending DataSourceAttribute where we can override a method that returns a DataTable or IEnumerable and have each DataRow available in TestContext?

[l3oferreira]

Hi @koskit , have you seen an extension that I created for this purpose? I think I can solve your problem. https://github.com/l3oferreira/MSTest.AzureDevOps

[koskit]

Hi @l3oferreira , yes I have, I tried the implementation but it is not for us since it requires a ton of modification in existing test cases.

The problem is the public void UnitTestMethod(**TestCaseData testCaseData**) part. We do not want parameters on our tests. And as I mentioned before, even if we used it, we would have to base.Parameters = testCaseData on each test manually, so our parent class BaseTest could have the data required.

Thank you for your contribution, the implementation seems solid.

The feature/addition to the framework I'm suggesting, has the following requirements: "Feed" the test case a DataTable that it can use to

1. Decide the iterations that will run according to the DataTable.Rows.Count()
2. Populate the TestContext.DataRow field with each row of the table.

I tried implementing a custom DataSourceAttribute, but the class DataSourceAttribute is sealed. I tried manually assigning a value to DataRow of TestContext but it's readonly/private-setter (which would get me half way there, since the test case would still have no iterations).

[koskit]

@jayaranigarg Can you provide with some follow up on this problem? Is it possible to have a custom attribute implementation for DataSource or maybe support "Microsoft.VisualStudio.TestTools.DataSource.TestCase"?

[smopuim]

Any updates on this issue?

[rhyous]

I notice this is up-for-grabs.

1. Is the current .NET Framework DataSourceAttribute code open source?
2. Does TestContext exist in MSTest2 and would it work if the attribute was coded, or is there more wiring missing than just the lack of this attribute?

[avibitran]

Hi all, THIS IS MY FIRST CONTRIBUTION. As a follower of MSTest, and with 14 yrs. of experience in Automation, I've come-up with a solution to extend DataSourceAttribute to fit the new Extensibility of MSTest. This is my solution, hope for feedback to fix whatever you'll find, hope it helps:

The Extensibility

1. to get the test configuration section connection from config file:

private Microsoft.VisualStudio.TestTools.UnitTesting.TestConfigurationSection GetTestConfigurationSection()
{
    return (Microsoft.VisualStudio.TestTools.UnitTesting.TestConfigurationSection)_config.Sections["Microsoft.VisualStudio.TestTools"];
}

2. Extract the ConnectionStringSettings using the connectionstring name:

public System.Configuration.ConnectionStringSettings GetConnectionString(string connectionStringName)
{
    System.Configuration.ConnectionStringSettingsCollection connectionString;

    connectionString = GetConnectionStringsSettingsSection();

    return connectionString[connectionStringName];
}

3. Using the previous methods, we're creating a new DataSourceAttribute containing the details of the connection extract from the connectionStringName from the config file:

public Microsoft.VisualStudio.TestTools.UnitTesting.DataSourceAttribute GetTestDataSource(string dataSourceSettingName)
{
    Microsoft.VisualStudio.TestTools.UnitTesting.DataSourceElement element = null;
    DataSourceAttribute dataSourceAttribute = null;

    TestConfigurationSection configurationSection = GetTestConfigurationSection();
    element = configurationSection.DataSources[dataSourceSettingName];

    if(element != null)
    {
        System.Configuration.ConnectionStringSettings connectionString = GetConnectionString(element.ConnectionString);
        dataSourceAttribute = new DataSourceAttribute(connectionString.ProviderName, connectionString.ConnectionString, element.DataTableName, (DataAccessMethod)Enum.Parse(typeof(DataAccessMethod), element.DataAccessMethod));
    }

    return dataSourceAttribute;
}

4. Finally we create a new Attribute that corresponds the the convention of MSTest Convention, and duplicate the functionality of the DataSourceAttribute:

using System;
using System.Collections.Generic;
using System.Data;
using System.Data.Common;
using System.Globalization;
using System.Linq;
using System.Reflection;

namespace Microsoft.VisualStudio.TestTools.UnitTesting
{
    [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
    public class DynamicDatasourceAttribute
        : Attribute, ITestDataSource
    {
        #region Fields
        public static readonly DataAccessMethod DefaultDataAccessMethod = DataAccessMethod.Random;
        public static readonly string DefaultProviderName = "System.Data.OleDb";
        private DataSourceAttribute _dataSource;
        #endregion Fields

        #region Ctor
        /// <summary>
        ///     Initializes a new instance of the DynamicDatasourceAttribute class. 
        ///     This instance will be initialized with a data provider and connection string associated with the setting name.
        /// </summary>
        /// <param name="dataSourceSettingName">The name of a data source found in the <microsoft.visualstudio.qualitytools> section in the ConfigurationManager configuration file.</param>
        public DynamicDatasourceAttribute(string dataSourceSettingName)
        {
            _dataSource = GetTestDataSource(dataSourceSettingName);
        }

        /// <summary>
        ///     Initializes a new instance of the DynamicDatasourceAttribute class.
        ///     This instance will be initialized with a connection string and table name. Specify connection string and data table to access OLEDB data source.
        /// </summary>
        /// <param name="connectionString">
        ///     Data provider specific connection string. 
        ///     WARNING: The connection string can contain sensitive data (for example, a password). 
        ///     The connection string is stored in plain text in source code and in the compiled assembly. 
        ///     Restrict access to the source code and assembly to protect this sensitive information.
        /// </param>
        /// <param name="tableName">The name of the data table.</param>
        public DynamicDatasourceAttribute(string connectionString, string tableName)
            : this(DefaultProviderName, connectionString, tableName, DefaultDataAccessMethod)
        { }

        /// <summary>
        ///     Initializes a new instance of the DataSourceAttribute class. 
        ///     This instance will be initialized with a data provider, connection string, data table and data access method to access the data source.
        /// </summary>
        /// <param name="providerInvariantName">Invariant data provider name, such as System.Data.SqlClient .</param>
        /// <param name="connectionString">
        ///     Data provider specific connection string. 
        ///     WARNING: The connection string can contain sensitive data (for example, a password). 
        ///     The connection string is stored in plain text in source code and in the compiled assembly. 
        ///     Restrict access to the source code and assembly to protect this sensitive information.</param>
        /// <param name="tableName">The name of the data table.</param>
        /// <param name="dataAccessMethod">Specifies the order to access data.</param>
        public DynamicDatasourceAttribute(string providerInvariantName, string connectionString, string tableName, DataAccessMethod dataAccessMethod)
        {
            _dataSource = new DataSourceAttribute(providerInvariantName, connectionString, tableName, dataAccessMethod);
        }
        #endregion Ctor

        #region Methods
        public IEnumerable<object[]> GetData(MethodInfo methodInfo)
        {
            List<object[]> rows = null;

            DbProviderFactory providerFactory = DbProviderFactories.GetFactory(_dataSource.ProviderInvariantName);

            using (DbConnection connection = providerFactory.CreateConnection())
            {
                connection.ConnectionString = _dataSource.ConnectionString;

                using (DbCommand command = connection.CreateCommand())
                {
                    command.CommandText = $"SELECT * FROM [{_dataSource.TableName}]";
                    command.Connection.Open();

                    using (DbDataReader reader = command.ExecuteReader())
                    {
                        if(reader.HasRows)
                        {
                            ParameterInfo[] parameters = methodInfo.GetParameters();
                            rows = new List<object[]>();

                            while (reader.Read())
                            {
                                List<object> row = new List<object>();

                                for(int i = 0; i < parameters.Length; i++)
                                {
                                    row.Add(Convert.ChangeType(reader[i], parameters[i].ParameterType));
                                }

                                rows.Add(row.ToArray());
                            }
                        }
                    }
                }
            }

            return rows;
        }

        public string GetDisplayName(MethodInfo methodInfo, object[] data)
        {
            return string.Format(CultureInfo.CurrentCulture, "{0}, Data: {1}", methodInfo.Name, string.Join(", ", data.AsEnumerable()));
        }

        #region Private Methods
        private string GetTestDescription(MethodInfo methodInfo)
        {
            string desc = String.Empty;
            Attribute descriptionAttribute;

            descriptionAttribute = methodInfo.GetCustomAttribute<TestMethodExAttribute>(true);

            if (descriptionAttribute != null)
            {
                desc = ((TestMethodExAttribute)descriptionAttribute).DisplayName;
            }
            else
            {
                descriptionAttribute = methodInfo.GetCustomAttribute<DescriptionAttribute>(false);

                if (descriptionAttribute != null)
                {
                    desc = ((DescriptionAttribute)descriptionAttribute).Description;
                }
            }

            return desc;
        }
        #endregion Private Methods
        #endregion Methods

        #region Properties
        /// <summary>
        /// Gets a value representing the data provider of the data source.
        /// </summary>
        /// <returns>
        /// The data provider name. If a data provider was not designated at object initialization, the default provider of System.Data.OleDb will be returned.
        /// </returns>
        public string ProviderInvariantName
        {
            get { return _dataSource.ProviderInvariantName; }
        }

        /// <summary>
        /// Gets a value representing the connection string for the data source.
        /// </summary>
        public string ConnectionString
        {
            get { return _dataSource.ConnectionString; }
        }

        /// <summary>
        /// Gets a value indicating the table name providing data.
        /// </summary>
        public string TableName
        {
            get { return _dataSource.TableName; }
        }

        /// <summary>
        /// Gets the method used to access the data source.
        /// </summary>
        ///
        /// <returns>
        /// One of the <see cref="DataAccessMethod"/> values. If the <see cref="DataSourceAttribute"/> is not initialized, this will return the default value <see cref="DataAccessMethod.Random"/>.
        /// </returns>
        public DataAccessMethod DataAccessMethod
        {
            get { return _dataSource.DataAccessMethod; }
        }

        /// <summary>
        /// Gets the name of a data source found in the &lt;microsoft.visualstudio.qualitytools&gt; section in the app.config file.
        /// </summary>
        public string DataSourceSettingName
        {
            get { return _dataSource.DataSourceSettingName; }
        }
        #endregion Properties
    }
}

The Implementation

1. The connectionStrings section in the config file is:

<connectionStrings>
    <add name="LoginTestsConnectionString"
        connectionString="Provider=Microsoft.ACE.OLEDB.12.0; Data Source=C:\LoginTests.xlsx; Extended Properties='Excel 12.0 Xml;HDR=YES';" providerName="System.Data.OleDb" />
</connectionStrings>

2. The datasource section in the config file is:

<Microsoft.VisualStudio.TestTools>
    <dataSources>
      <add name="LoginTestsDS"
      connectionString="LoginTestsConnectionString"
      dataTableName="LoginTest$"
      dataAccessMethod="Sequential"/>
    </dataSources>
  </Microsoft.VisualStudio.TestTools>

3. The excel file named: LoginTests.xlsx should contain a sheet named: LoginTest, with first column (ROW_INDEX) containing int values and second column (PARAM1) containing boolean values (TRUE, FALSE). This is corresponding to the testcase parameters 4. An example UnitTestCase should look like this:

[DataTestMethod]
        [Description("Description of UTF.TestTools.Testing.UnitTest1.TestMethod14")]
        [TestCategory(TestType.INTEGRATION_TEST)]
        [TestCategory(TestStatus.COMPLETED)]
        [TestCategory(TestOwner.ABitran)]
        [DynamicDatasource("LoginTestsDS")]
        public void TestMethod14(int rowIndex, bool input)
        {
            StepInfo step;

            Report.Test.AddStep("Running Test", $"Running {TestContext.TestName}");

            Report.Test.AddStep("Step 2", "The step passed", "The step did pass", StepStatusEnum.Pass);

            Report.Test.AddStep("Step 3", "The step passed", "The step did pass", StepStatusEnum.Warning);

            step = Report.Test.AddStep("Step 4", "The step passed");
            Assert.That.ReportIsTrue(input, step, "The step didn't pass", "The step did pass");
        }

[giannispatrick]

Guys any update to this? We want to be upgrade to MSTest2 but this is holding us back.

[Evangelink]

Hi @giannispatrick,

There is currently no plan to bring this feature to MSTest v3.