reqnroll / Reqnroll

Open-source Cucumber-style BDD test automation framework for .NET.
https://reqnroll.net
BSD 3-Clause "New" or "Revised" License
310 stars 32 forks source link

Parallel scenario tests fail #114

Closed reservoir-dogs closed 2 months ago

reservoir-dogs commented 2 months ago

Reqnroll Version

1.0.1

Which test runner are you using?

MSTest

Test Runner Version Number

VS 17.9.2

.NET Implementation

.NET 6.0

Test Execution Method

Visual Studio Test Explorer

Content of reqnroll.json configuration file

No response

Issue Description

It seems ScenarioContext did not create on each parallel test

File Scenario.feature

Feature: Scenario

Scenario: Scenario 1
    When Initialize
    Then Assert

Scenario: Scenario 2
    When Initialize
    Then Assert

File ScenarioStepBinding

namespace ReqnrollProject1.StepDefinitions
{
    [Binding]
    public class ScenarioStepDefinitions
    {
        private readonly ScenarioContext scenarioContext;

        public ScenarioStepDefinitions(ScenarioContext scenarioContext)
        {
            this.scenarioContext = scenarioContext;
            Console.WriteLine("Constructor");
        }

        [BeforeScenario]
        public void Initialize()
        {
            scenarioContext.Add("Id", Guid.NewGuid());
        }

        [When("Initialize")]
        public void WhenInitialize()
        {
            Console.WriteLine("When: " + scenarioContext["Id"]);
        }

        [Then("Assert")]
        public void ThenAssert()
        {
            Console.WriteLine("Then: " + scenarioContext["Id"]);
        }
    }
}

File .runsettings

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <MSTest>
        <Parallelize>
            <Workers>0</Workers>
            <Scope>MethodLevel</Scope>
        </Parallelize>
    </MSTest>
</RunSettings>

Results test :

If I parallelize tests, then tests fail

dotnet test --settings:.\.runsettings
  Identification des projets à restaurer...
  Tous les projets sont à jour pour la restauration.
  ReqnrollFeatureFiles: Features\Scenario.feature
  -> Using default config
  ReqnrollGeneratedFiles: Features\Scenario.feature.cs
  ReqnrollProject1 -> C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\bin\Debug\net6.0\ReqnrollProject1.dll
Série de tests pour C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\bin\Debug\net6.0\ReqnrollProject1.dll (.NETCoreApp,Version=v6.0)
Outil en ligne de commande d'exécution de tests Microsoft (R), version 17.9.0 (x64)
Copyright (c) Microsoft Corporation. Tous droits réservés.

Démarrage de l'exécution de tests, patientez...
Au total, 1 fichiers de test ont correspondu au modèle spécifié.
  Échoué Scenario2 [30 ms]
  Message d'erreur :
   Test method ReqnrollProject1.Features.ScenarioFeature.Scenario2 threw exception:
System.ArgumentException: An item with the same key has already been added. Key: Id
  Arborescence des appels de procédure :
      at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at ReqnrollProject1.StepDefinitions.ScenarioStepDefinitions.Initialize() in C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\StepDefinitions\ScenarioStepDefinitions.cs:line 17
--- End of stack trace from previous location ---
   at Reqnroll.Bindings.BindingInvoker.InvokeBindingAsync(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, DurationHolder durationHolder)
   at Reqnroll.Infrastructure.TestExecutionEngine.InvokeHookAsync(IAsyncBindingInvoker invoker, IHookBinding hookBinding, HookType hookType)
   at Reqnroll.Infrastructure.TestExecutionEngine.FireEventsAsync(HookType hookType)
   at Reqnroll.Infrastructure.TestExecutionEngine.FireEventsAsync(HookType hookType)
   at Reqnroll.Infrastructure.TestExecutionEngine.FireScenarioEventsAsync(HookType bindingEvent)
   at Reqnroll.Infrastructure.TestExecutionEngine.OnScenarioStartAsync()
   at Reqnroll.Infrastructure.TestExecutionEngine.OnAfterLastStepAsync()
   at Reqnroll.TestRunner.CollectScenarioErrorsAsync()
   at Reqnroll.TestRunner.CollectScenarioErrorsAsync()
   at ReqnrollProject1.Features.ScenarioFeature.ScenarioCleanupAsync()
   at ReqnrollProject1.Features.ScenarioFeature.ScenarioCleanupAsync()
   at ReqnrollProject1.Features.ScenarioFeature.Scenario1() in C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\Features\Scenario.feature:line 5
   at ReqnrollProject1.Features.ScenarioFeature.Scenario2() in C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\Features\Scenario.feature:line 9

  Messages de sortie standard :
 -> warning: The previous ScenarioContext was not disposed.
 When Initialize
 -> skipped because of previous errors
 Then Assert
 -> skipped because of previous errors

 TestContext Messages:
 -> warning: The previous ScenarioContext was not disposed.
 When Initialize
 -> skipped because of previous errors
 Then Assert
 -> skipped because of previous errors

  Échoué Scenario1 [30 ms]
  Message d'erreur :
   Test method ReqnrollProject1.Features.ScenarioFeature.Scenario1 threw exception:
System.ArgumentException: An item with the same key has already been added. Key: Id
  Arborescence des appels de procédure :
      at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at ReqnrollProject1.StepDefinitions.ScenarioStepDefinitions.Initialize() in C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\StepDefinitions\ScenarioStepDefinitions.cs:line 17
--- End of stack trace from previous location ---
   at Reqnroll.Bindings.BindingInvoker.InvokeBindingAsync(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, DurationHolder durationHolder)
   at Reqnroll.Infrastructure.TestExecutionEngine.InvokeHookAsync(IAsyncBindingInvoker invoker, IHookBinding hookBinding, HookType hookType)
   at Reqnroll.Infrastructure.TestExecutionEngine.FireEventsAsync(HookType hookType)
   at Reqnroll.Infrastructure.TestExecutionEngine.FireEventsAsync(HookType hookType)
   at Reqnroll.Infrastructure.TestExecutionEngine.FireScenarioEventsAsync(HookType bindingEvent)
   at Reqnroll.Infrastructure.TestExecutionEngine.OnScenarioStartAsync()
   at Reqnroll.Infrastructure.TestExecutionEngine.OnAfterLastStepAsync()
   at Reqnroll.TestRunner.CollectScenarioErrorsAsync()
   at Reqnroll.TestRunner.CollectScenarioErrorsAsync()
   at ReqnrollProject1.Features.ScenarioFeature.ScenarioCleanupAsync()
   at ReqnrollProject1.Features.ScenarioFeature.ScenarioCleanupAsync()
   at ReqnrollProject1.Features.ScenarioFeature.Scenario1() in C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\Features\Scenario.feature:line 5
   at ReqnrollProject1.Features.ScenarioFeature.Scenario2() in C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\Features\Scenario.feature:line 9

  Messages de sortie standard :
 -> Loading plugin C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\bin\Debug\net6.0\Reqnroll.MSTest.ReqnrollPlugin.dll
 -> Loading plugin C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\bin\Debug\net6.0\ReqnrollProject1.dll
 -> Using default config
 Constructor
 When Initialize
 -> skipped because of previous errors
 Then Assert
 -> skipped because of previous errors

 TestContext Messages:
 When Initialize
 -> skipped because of previous errors
 Then Assert
 -> skipped because of previous errors

Échoué!  - échec :     2, réussite :     0, ignorée(s) :     0, total :     2, durée : 408 ms - ReqnrollProject1.dll (net6.0)

But if I didn't parallelize tests, then tests are success.

dotnet test
  Identification des projets à restaurer...
  Tous les projets sont à jour pour la restauration.
  ReqnrollFeatureFiles: Features\Scenario.feature
  -> Using default config
  ReqnrollGeneratedFiles: Features\Scenario.feature.cs
  ReqnrollProject1 -> C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\bin\Debug\net6.0\ReqnrollProject1.dll
Série de tests pour C:\Projets\POC\ReqnrollProject1\ReqnrollProject1\bin\Debug\net6.0\ReqnrollProject1.dll (.NETCoreApp,Version=v6.0)
Outil en ligne de commande d'exécution de tests Microsoft (R), version 17.9.0 (x64)
Copyright (c) Microsoft Corporation. Tous droits réservés.

Démarrage de l'exécution de tests, patientez...
Au total, 1 fichiers de test ont correspondu au modèle spécifié.

Réussi!  - échec :     0, réussite :     2, ignorée(s) :     0, total :     2, durée : 419 ms - ReqnrollProject1.dll (net6.0)

Steps to Reproduce

ReqnrollProject1.zip

Link to Repro Project

No response

gasparnagy commented 2 months ago

Thx @reservoir-dogs. Unfortunately Reqnroll (just like SpecFlow) does not support method-level parallelization yet. So it is "by design".

We would like to provide support for that in the close future though. Could you please post this as a feature idea here: https://github.com/orgs/reqnroll/discussions/categories/ideas

I will close this as "by design", but you can track the feature implementation progress on the discussion item you create.