Collection was modified; enumeration operation may not execute.

[gao-artur]

Saw this error in OpenTelemetry.Instrumentation.Hangfire.Tests.

System.InvalidOperationException Collection was modified; enumeration operation may not execute. at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource) at System.Collections.Generic.List1.Enumerator.MoveNextRare() at System.Linq.Enumerable.All[TSource](IEnumerable1 source, Func2 predicate) at OpenTelemetry.Instrumentation.Hangfire.Tests.HangfireInstrumentationJobFilterAttributeTests.<WaitJobProcessedAsync>d__8.MoveNext() in C:\github\opentelemetry-dotnet-contrib\test\OpenTelemetry.Instrumentation.Hangfire.Tests\HangfireInstrumentationJobFilterAttributeTests.cs:line 181 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at OpenTelemetry.Instrumentation.Hangfire.Tests.HangfireInstrumentationJobFilterAttributeTests.<Should_Create_Activity_With_Status_Error_When_Job_Failed>d__3.MoveNext() in C:\github\opentelemetry-dotnet-contrib\test\OpenTelemetry.Instrumentation.Hangfire.Tests\HangfireInstrumentationJobFilterAttributeTests.cs:line 58 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Xunit.Sdk.TestInvoker1.<>cDisplayClass48_0.<b_1>d.MoveNext() in //src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 276 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Xunit.Sdk.ExecutionTimer.d4.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Xunit.Sdk.ExceptionAggregator.d_9.MoveNext() in //src/xunit.core/Sdk/ExceptionAggregator.cs:line 90

It happens in WaitJobProcessedAsync method

private async Task WaitJobProcessedAsync(string jobId, int timeToWaitInSeconds)
{
    var timeout = DateTime.Now.AddSeconds(timeToWaitInSeconds);
    string[] states = new[] { "Enqueued", "Processing" };
    JobDetailsDto jobDetails;
    while (((jobDetails = this.hangfireFixture.MonitoringApi.JobDetails(jobId)) == null ||
            jobDetails.History.All(h => states.Contains(h.StateName))) &&
           DateTime.Now < timeout)
    {
        await Task.Delay(500);
    }
}

From my understanding, the MonitoringApi.JobDetails(jobId).History returns the reference to the list instance used as in-memory storage. If this list changes during the enumeration, the caller will get the mentioned error.

Instead, the data snapshot should be returned. No modification should be allowed during the snapshot creation.