Random exception during test execution

[kemsky]

I'm submitting a ...

• [x] bug report
• [ ] feature request
• [ ] support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

[22:05:51][dotnet test] The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
[22:05:51][dotnet test]    at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
[22:05:51][dotnet test]    at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
[22:05:51][dotnet test]    at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
[22:05:51][dotnet test]    at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
[22:05:51][dotnet test]    at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
[22:05:51][dotnet test]    at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
[22:05:51][dotnet test]    at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
[22:05:51][dotnet test]    at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value)
[22:05:51][dotnet test]    at Newtonsoft.Json.JsonSerializer.Serialize(TextWriter textWriter, Object value)
[22:05:51][dotnet test]    at Allure.Net.Commons.Writer.FileSystemResultsWriter.Write(Object allureObject, String fileSuffix)
[22:05:51][dotnet test]    at Allure.Net.Commons.Writer.FileSystemResultsWriter.Write(TestResultContainer testResult)
[22:05:51][dotnet test]    at Allure.Net.Commons.AllureLifecycle.WriteTestContainer()
[22:05:51][dotnet test]    at Allure.NUnit.Core.AllureNUnitHelper.StopTestContainer()
[22:05:51][dotnet test]    at Allure.NUnit.AllureNUnitAttribute.<>c__DisplayClass2_0.<AfterTest>b__0()
[22:05:51][dotnet test]    at Allure.NUnit.AllureNUnitAttribute.RunHookInRestoredAllureContext(ITest test, Action action)
[22:05:51][dotnet test]    at Allure.NUnit.AllureNUnitAttribute.AfterTest(ITest test)
[22:05:51][dotnet test]    at NUnit.Framework.Internal.Commands.TestActionItem.AfterTest(ITest test)
[22:05:51][dotnet test]    at NUnit.Framework.Internal.Commands.AfterTestActionCommand.<>c__DisplayClass0_0.<.ctor>b__0(TestExecutionContext context)
[22:05:51][dotnet test]    at NUnit.Framework.Internal.Commands.AfterTestCommand.Execute(TestExecutionContext context)
[22:05:51][dotnet test]    at NUnit.Framework.Internal.Execution.CompositeWorkItem.PerformOneTimeTearDown()
[22:05:51][dotnet test]    at NUnit.Framework.Internal.Execution.CompositeWorkItem.OneTimeTearDownWorkItem.Execute()
[22:05:51][dotnet test]    at NUnit.Framework.Internal.Execution.TestWorker.TestWorkerThreadProc()
[22:05:51][dotnet test]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[22:05:51][dotnet test] --- End of stack trace from previous location ---
[22:05:51][dotnet test]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

If the current behavior is a bug, please provide the steps to reproduce and, if possible, a minimal demo of the problem

Please tell us about your environment:

• OS Windows
• .NET framework .NET 8
• Test framework: nunit@4.1.0
• Allure adapter: allure-nunit@2.12.0

[delatrie]

Hi, @kemsky !

Do you define Allure fixtures with AllureBefore or AllureAfter ? If so, do you define steps in them? Do fixtures or steps have parameters? If yes, what types do you use?

[kemsky]

@delatrie, there is no AllureBefore or AllureAfter in my tests, also there are no steps at all. Some tests have TestCase attributes, types are primitive, numbers, strings, enums.

SetUpFixture :

[SetUpFixture]
public class TestInitializer
{
    [OneTimeSetUp]
    public void Setup()
    {
        CultureContext.SetThreadCulture();
    }

    [OneTimeTearDown]
    public void Teardown()
    {
    }
}

A couple of tests have [SetUp] attribute.

Many tests are marked with [Parallelizable(ParallelScope.All)].

[delatrie]

Ok, thank you for your quick response. How often does the exception occur?

[kemsky]

Approximately 1 out of 10 runs fails

[kemsky]

There are ~500 tests.

[delatrie]

My first impression is that there is a data race somewhere in our code. Does turning off the parallelization help?