Closed gasparnagy closed 7 months ago
@livioc Reqnroll adds an additional string[] parameter to the method to know about the tags (regardless of whether the test framework supports it) and that had to be provided.
The rest of the issues seem to be related to how we do the testing: we let the tests run and then parse the TRX file to get the results. But MsTest generates the TRX file in a slightly different format in case of DataRow, so that causes the problems. I have a working env now, I can try to debug the rest.
Oh God. This was a tricky issue. Somehow the code we generated was not working in case there was only one column in the examples table with MsTest v2.*...
Locally it is passing now. Let's hope it works.
But I had to learn that the local testability is really a pain, especially for generator changes. Need to improve this (https://github.com/orgs/reqnroll/discussions/38).
Hi @gasparnagy , today I have done some tests on a new .net48 test project but I still have problem.
I created the nuget packages with .\build.ps1 -Configuration Release
on https://github.com/reqnroll/Reqnroll/commit/e999ecfd87517a4d91c841ade7571fb612f4fb8e.
Then I generated the test code using <PackageReference Include="Reqnroll.MsTest" Version="1.0.2-local" />
with this gherkin
Scenario Outline: Simple scenario
Given there is <count> <color> items
Examples:
| count | color |
| 0 | blue |
| 0 | black |
When I run the generated test code
[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
[Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("Simple scenario")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "Calculator")]
[Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("0", "blue")]
[Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("0", "black")]
public async System.Threading.Tasks.Task SimpleScenario(string count, string color, string[] exampleTags)
{
string[] tagsOfScenario = exampleTags;
System.Collections.Specialized.OrderedDictionary argumentsOfScenario = new System.Collections.Specialized.OrderedDictionary();
argumentsOfScenario.Add("count", count);
argumentsOfScenario.Add("color", color);
Reqnroll.ScenarioInfo scenarioInfo = new Reqnroll.ScenarioInfo("Simple scenario", null, tagsOfScenario, argumentsOfScenario, featureTags);
#line 15
this.ScenarioInitialize(scenarioInfo);
#line hidden
if ((TagHelper.ContainsIgnoreTag(tagsOfScenario) || TagHelper.ContainsIgnoreTag(featureTags)))
{
testRunner.SkipScenario();
}
else
{
await this.ScenarioStartAsync();
#line 16
await testRunner.GivenAsync(string.Format("there is {0} {1} items", count, color), ((string)(null)), ((Reqnroll.Table)(null)), "Given ");
#line hidden
}
await this.ScenarioCleanupAsync();
}
I got the exception
SimpleScenario (0,black)
Source: Calculator.feature line 15
Duration: 71 ms
Message:
Test method ReqnrollProject1.Features.CalculatorFeature.SimpleScenario threw exception:
System.Reflection.TargetParameterCountException: Parameter count mismatch.
TestCleanup method ReqnrollProject1.Features.CalculatorFeature.TestTearDownAsync threw exception. System.NullReferenceException: Object reference not set to an instance of an object..
Stack Trace:
RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
TestCleanup Stack Trace
<OnScenarioEndAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
TaskAwaiter.GetResult()
<OnScenarioEndAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
TaskAwaiter.GetResult()
<TestTearDownAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Standard Output:
-> Loading plugin C:\dev\ExtGitHub\Reqnroll-PersonalTest\ReqnrollProject1\bin\Debug\net48\Reqnroll.MSTest.ReqnrollPlugin.dll
-> Loading plugin C:\dev\ExtGitHub\Reqnroll-PersonalTest\ReqnrollProject1\bin\Debug\net48\ReqnrollProject1.dll
-> Using default config
-> warning: The previous ScenarioContext was already disposed.
TestContext Messages:
-> warning: The previous ScenarioContext was already disposed.
Are you sure that we must pass the addition tags array parameter? Can't we just ignore them?
@livioc The current test system is not working properly. Actually you are not testing with the version you have built (you can see it because the [Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("0", "black")]
has only two parameters, but it should have 3.
The problem is that the package has been cached to your C:\Users\<your user>\.nuget\packages
folder and therefore it does not matter if you compile it locally, you will still test with the previous one (I work on a solution in #62).
What you can do is to manually delete all reqnroll related packages from C:\Users\<your user>\.nuget\packages
and then try again.
@gasparnagy thanks, I have done same tests, this scenaraio doens't work with mstest
The generated code has a different number of parameters
[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
[Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("Another scenario")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "Calculator")]
[Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("1", "", new string[] {
"mytag"})]
[Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("2", null)]
public async System.Threading.Tasks.Task AnotherScenario(string count, string notUsed6248, string[] exampleTags)
{
string[] tagsOfScenario = exampleTags;
System.Collections.Specialized.OrderedDictionary argumentsOfScenario = new System.Collections.Specialized.OrderedDictionary();
argumentsOfScenario.Add("count", count);
Reqnroll.ScenarioInfo scenarioInfo = new Reqnroll.ScenarioInfo("Another scenario", null, tagsOfScenario, argumentsOfScenario, featureTags);
#line 25
this.ScenarioInitialize(scenarioInfo);
#line hidden
if ((TagHelper.ContainsIgnoreTag(tagsOfScenario) || TagHelper.ContainsIgnoreTag(featureTags)))
{
testRunner.SkipScenario();
}
else
{
await this.ScenarioStartAsync();
#line 26
await testRunner.GivenAsync(string.Format("there is {0} blue items", count), ((string)(null)), ((Reqnroll.Table)(null)), "Given ");
#line hidden
}
await this.ScenarioCleanupAsync();
}
but I don't understand this, this code works with a .net48 test project
[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
[Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("Another scenario")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "Calculator")]
[Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("1", new string[] {
"mytag"})]
[Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("2", null)]
public async System.Threading.Tasks.Task AnotherScenario(string count, string[] exampleTags)
{
string[] tagsOfScenario = exampleTags;
System.Collections.Specialized.OrderedDictionary argumentsOfScenario = new System.Collections.Specialized.OrderedDictionary();
argumentsOfScenario.Add("count", count);
// ....
}
@livioc ah yes, thx. this is a bug and I know why. I'll fix. (I have no idea why this was working on .net48)
@livioc I fixed it now. I could only do the related test in my new "System Test" concept in #62 (a214d47), but that will be merged at some point.
If you have a little more time, please retest it. You can also have a look at my system test and give feedback if such testing would be useful/efficient for generator changes.
(This is a redo of PR #52 by @livioc, to be able to fix the generation issues.)
Types of changes
Checklist: