Collection was modified; enumeration operation may not execute when trying to resolve controller

[sachinmani]

I sporadically see this exception in my .net core API

Collection was modified; enumeration operation may not execute. - Collection was modified; enumeration operation may not execute.

at SimpleInjector.InstanceProducer.GetInstance()
at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)`

Has anyone come across this problem and is there any fix for this?

[dotnetjunkie]

This has likely something to do with custom code (your code) that runs during the construction of your controller (and its dependencies), but without a repro it's hard to say anything of value (seems like a concurrency issue though). Can you provide a Minimal, Complete, and Verifiable example that reproduces the problem?

[sachinmani]

public void ConfigureServices(IServiceCollection services)
{
    services.AddRouting(options =>
    {
        options.LowercaseQueryStrings = true;
        options.LowercaseUrls = true;
    });
    services.AddControllers(options => {
        options.Filters.Add(typeof(AddIdsToContextFilter)); })
        // Configure options to keep responses consistent
        .ConfigureApiBehaviorOptions(options =>
        {
            options.SuppressMapClientErrors = true;
            options.SuppressModelStateInvalidFilter = true;
        })
        .AddNewtonsoftJson(options =>
        {
            options.SerializerSettings.NullValueHandling = NullValueHandling.Include;
        });

    services.AddSimpleInjector(_container, options =>
    {
        options.AddAspNetCore().AddControllerActivation();

        options.AddLogging();
    });

    IocHelper.RegisterCommonBinding(_container, Configuration);
}

public void Configure(IApplicationBuilder app, IHostEnvironment env)
{
    app.UseSimpleInjector(_container);

    app.UseCorrelationId(new CorrelationIdOptions
    {
        Header = "X-Correlation-ID",
        UseGuidForCorrelationId = true,
        UpdateTraceIdentifier = false
    });

    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }
    else
    {
        // App.UseHsts() doesn't seem to be working, but we aren't doing anything clever, so lets just add it manually
        // We'll add these headers when it's not a dev environment to make developer's lives easier
        // I haven't included everything here:
        // Access-Control headers - Not sure what CORS stuff we need so I simply haven't enabled CORS here
        // Removal of Kestrel Server header is done in Program.cs
        app.Use((context, next) =>
        {
            context.Response.Headers.Add("X-Frame-Options", "deny");
            context.Response.Headers.Add("X-Content-Type-Options", "nosniff");
            context.Response.Headers.Add("Strict-Transport-Security", "max-age=31557600; includeSubDomains");
            context.Response.Headers.Add("Cache-Control", "no-store");
            context.Response.Headers.Add("Pragma", "no-cache");
            return next();
        });
    }

    // Rest of the configure stuff (wiring up middleware etc.) is done in Bootstrap
    app.UseRouting();
    app.UseAuthentication();
    app.UseAuthorization();
    app.UseMiddleware<QuoteMiddleware>(_container);

    var logger = _container.GetInstance<ILogger<Startup>>();
    BootStrap.Configure(app, env, logger);

    app.UseEndpoints(endpoints => 
    {
        endpoints.MapHealthChecks(Infrastructure.Constants.HealthCheckRoute, new HealthCheckOptions()
        {
            ResponseWriter = HealthCheck.WriteResponse
        });
        endpoints.MapControllers();
    });

    _container.Verify();
}

Looking at it doesn't look like any concurrency issue. Also the issue is not reproducable and happens so randomly. In our load test we saw 10 instances of it out of 100,000 requests

[dotnetjunkie]

I can't find anything odd about the code you posted. My advise is the following: find out which controller it is that is being resolved when the exception happens. You should be able to find this out by looking at the information (url) of the failing request. Review the controller and its dependencies. Take a close look at any code that executes during initialization; typically constructors and any code they call. You will likely find some code that modifies a collection.

Also take a look whether there any inner exceptions. If there are any, please post all their relevant details here. They likely contain much more information.

Good luck.

[dotnetjunkie]

Hi @sachinmani,

Did you have any luck in finding the cause of your problem? Is there anything I can help you with?