microsoft / testfx

MSTest framework and adapter
MIT License
720 stars 253 forks source link

Not able to get datarow attribute value in TestContext and initialization. #1355

Open Baccanno opened 1 year ago

Baccanno commented 1 year ago

This exact same issue has been raised before here and closed : https://github.com/microsoft/testfx/issues/305 There is no way to retrieve the test DataRowAttribute data in test initialization.

People might have different reason to do so such as simply being able to create a Log file for each case.

So what is the solution or the workaround ? The only thing I can think of is cache the method arguments through an attribute inheriting from DataTestMethodAttribute and retrieve them in the init method. But it would be highly dependant on how the Execute method is operating. It would only work if tests arguments could be queued/dequeued sequentially.

We're using version 14.0.0.0

image

Evangelink commented 1 year ago

Hi @Baccanno! Thanks for the suggestion. I have looked at the linked ticket and there are multiple requests and confusion in there so I'd like to be sure of what you are trying to achieve. Do you want to be able to access to the test parameters during the test initialization (TestInitializeAttribute) and/or test cleanup (TestCleanupAttribute)? If so could you please explain your reasoning there (basically what do you want to be able to achieve doing so)?

To give some clarifications about the linked ticket:

Baccanno commented 1 year ago

Hi @Evangelink Thanks for taking the time look at it. Indeed I read the discussion and saw the conclusion about the DataRow property naming. Yeah basically you said it. When in the TestInitializeAttribute and TestCleanupAttribute we'd like to have the info of the test parameters.

Basically we're creating an individual log file in most of our TestInitializeAttribute and TestCleanupAttribute methods. Right now for multiple data test methods I need to postpone the common test initialisation in the test method itself, plus I need to adapt other test method to avoid duplicate initialization.

I'd like to be able to perform the follwing


[TestInitialize]
public void TestInit() {
    //CreateLogFile With test params
}

[DataTestMethod]
[DataRow(0)]
[DataRow(1)]
MyTestA(int a)
{
  //Do test
}

MyTestB(){
  //Do test
}

Today for it to work with DataTestMethod attribute I need to do this, which defeats the purpose of having a TestInit attribute. And ask to refactor test classes wherever we add DataTestMethod tests


//[TestInitialize] 
public void TestInit(param object[] testParams) {
    //CreateLogFile With the test params 
}

[DataTestMethod]
[DataRow(0)]
[DataRow(1)]
MyTestA(int a)
{
  TestInit(a);
  //Do test
}

MyTestB(){
 TestInit();
  //Do test
}
Evangelink commented 1 year ago

Thanks for taking the time to give a good example of what you would expect. We will look into the possible solutions and reply back with the options.

Stay tuned.

Mertsch commented 1 year ago

I always have the same issue. I want TestContext.DataRow to be what it should be :-) If that is not possible maybe we can get a TestContext.TestData? And maybe a TestDataAttribute which does exactly what DataRow does today, but to avoid naming confusion.

Or maybe we can have the TestData as value in TestContext.TestData. Would make life so much easier!

I am not in favor of TestInitialize with object[] args. I think that is a too generic approach and might cause problem with overloads in the future. And we do have TestContext in TestInitialize.

Evangelink commented 1 year ago

It feels like something that should be generic enough to also handle DynamicData but also probably provide other information to user. I am not sure how far we should go (for example we could simply expose the test method object we have and that would contain the method name, signature, display name, args...).

builder-main commented 1 year ago

I always have the same issue. I want TestContext.DataRow to be what it should be :-) If that is not possible maybe we can get a TestContext.TestData? And maybe a TestDataAttribute which does exactly what DataRow does today, but to avoid naming confusion.

Or maybe we can have the TestData as value in TestContext.TestData. Would make life so much easier!

I am not in favor of TestInitialize with object[] args. I think that is a too generic approach and might cause problem with overloads in the future. And we do have TestContext in TestInitialize.

I'm not sure you're mentioning the latest because of my example, I was indeed showing what I need to do, but don't really want to, to retrieve the test parameters vs what I'd like to do that is as you perfectly said :

I want TestContext.DataRow to be what it should be :-)

@Evangelink I totally agree with the DynamicData route as well, we have a bunch of tests using it and would need a proper way to access a simily test Index, and display name as well as a the arguments.

SimonCropp commented 2 days ago

@Evangelink any movement on exposing test arguments for all scenarios? i would really like to use this feature in Verify

Evangelink commented 1 day ago

Sadly no progress on that front but given that it's taking more time than anticipated to refactor the test context objects, it seems like we could add a first version for 3.7