reportportal / agent-net-specflow

Report Portal agent for SpecFlow
Apache License 2.0
10 stars 4 forks source link

Skipped Scenarios are not displayed in RP when ignoring it at runtime #70

Closed iRetlaWM closed 3 years ago

iRetlaWM commented 3 years ago

I had 2 behaviors:

  1. When I use the @ignore tag used for Specflow to skip an scenario this does not appears in the RP, but this is displayed in the console log running with dotnet image So basically if I had 10 scenarios and 2 of them are with this tag ... we have only 8 scenarios displayed in RP

  2. Modifying the status of the test case during runtime I created @skip tag 1, and I create a BeforeScenario hook in order that if the scenario contains the tag skip I can modify during runtime the scenario, but the test case is marked as Failed instead of Skipped in RP, this is the piece of code that I created for that:

    [BeforeScenario]
    public void BeforeScenario(ScenarioContext context)
    {
      if (context.ScenarioInfo.ScenarioAndFeatureTags.Contains("skip"))
      {
          _unitTestRuntimeProvider.TestIgnore("This scenario is always skipped");
      }
    }

This raises the following error:

NUnit.Framework.IgnoreException: This scenario is always skipped
   at NUnit.Framework.Assert.Ignore(String message, Object[] args)
   at NUnit.Framework.Assert.Ignore(String message)
   at TechTalk.SpecFlow.NUnit.SpecFlowPlugin.NUnitRuntimeProvider.TestIgnore(String message)

I post this in RP slack channel and @nvborisenko help me to find a workaround to mark the Scenario as Skipped in RP using this code:

 private static void ReportPortalAddin_BeforeScenarioFinished(object sender, TestItemFinishedEventArgs e)
        {
           if (e.ScenarioContext.TestError != null && 
                e.ScenarioContext.ScenarioInfo.ScenarioAndFeatureTags.Contains("skip"))            
                e.FinishTestItemRequest.Status = Status.Skipped;
        }

Maybe you can address this behavior to avoid this piece of code and catch up the IgnoreException and mark the test case as Skipped

nvborisenko commented 3 years ago
  1. Known issue #55
  2. Looks like this situation can be handled by the agent. If TestError contains well known exceptions like NUnit.IgnoreException then treat scenario as skipped instead of failed. Don't forget about MSTest and xUnit.