Problems with scoped lifetime services (AddScoped)

[heikkilamarko]

Hi,

We are using durable functions with dependency injection. In our Startup.cs file, we register our dependencies as follows:

services.AddScoped<IMyService>(serviceProvider => new MyService(...))

In our activity function, we are using normal constructor injection to get IMyService instances.

The problem is that even if we are using AddScoped for registering the service, during the activity function run, each class that asks for the service, gets a different instance of IMyService. This breaks our app logic, because IMyService users won't see each other's changes.

As a workaround to the earlier Azure Functions runtime DI issues, we had the following pinning in place FUNCTIONS_EXTENSION_VERSION = 2.0.12673.0. The pinned version is not supported anymore by Azure, so our function DI is now broken again.

[heikkilamarko]

I find it hard to believe this is by design. If it is so, documentation should state that very clearly.

[heikkilamarko]

The problem is not durable function specific. Same happens with normal TimerTrigger functions also.

[heikkilamarko]

I made more experiments and it seem that the DI problem occurs only when I have used services.AddHttpClient() to register HttpClients. If I comment out services.AddHttpClient() calls, AddScoped works as expected.

[dhanapalschandran]

Hi @heikkilamarko I am also facing the same issue, when this issue happens last week I have pinned to older runtime version, then it started working, now again breaking. I agree it's not an issue with durable function. Even I am using the same DI pattern for HttpClients, Do we have any workaround for httpclient register or waiting for team to provide global fix

[unt1tled]

@dhanapalschandran Same thing on my side. Could it be breaking again due to #5096?

[fabiocav]

@heikkilamarko can you please share repro code? Is this happening with factory registrations only? Some more details/repro will ensure we're looking exactly at the issue you're running into.

@unt1tled / @dhanapalschandran if you can share the code you're having issues with, we can validate with your scenarios as well.

Thanks!

[heikkilamarko]

Hi. See below for a repro code (see the comment block in the function code):

using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using System.Net.Http;

[assembly: FunctionsStartup(typeof(Demo.App.Startup))]

namespace Demo.App
{
    public class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            var services = builder.Services;

            services.AddHttpClient<MyHttpClient>();

            services
                .AddScoped<IMyService, MyService>()
                .AddScoped<IMyServiceSetter, MyServiceSetter>()
                .AddScoped<IMyServiceGetter, MyServiceGetter>();
        }
    }

    public class MyHttpClient
    {
        private readonly HttpClient httpClient;

        public MyHttpClient(HttpClient httpClient)
        {
            this.httpClient = httpClient;
        }
    }

    public interface IMyService
    {
        void Set(int value);

        int Get();
    }

    public class MyService : IMyService
    {
        private int value = 0;

        public int Get()
        {
            return value;
        }

        public void Set(int value)
        {
            this.value = value;
        }
    }

    public interface IMyServiceSetter
    {
        void SetValue(int value);
    }

    public class MyServiceSetter : IMyServiceSetter
    {
        private readonly IMyService myService;
        private readonly IMyServiceGetter myServiceGetter;

        public MyServiceSetter(IMyService myService, IMyServiceGetter myServiceGetter)
        {
            this.myService = myService;
            this.myServiceGetter = myServiceGetter;
        }

        public void SetValue(int value)
        {
            myService.Set(value);
        }
    }

    public interface IMyServiceGetter
    {
        int GetValue();
    }

    public class MyServiceGetter : IMyServiceGetter
    {
        private readonly IMyService myService;
        private readonly MyHttpClient myHttpClient;

        public MyServiceGetter(IMyService myService, MyHttpClient myHttpClient)
        {
            this.myService = myService;
            this.myHttpClient = myHttpClient;
        }

        public int GetValue()
        {
            return myService.Get();
        }
    }

    public class DemoFunction
    {
        private readonly IMyServiceSetter myServiceSetter;
        private readonly IMyServiceGetter myServiceGetter;

        public DemoFunction(
            IMyServiceSetter myServiceSetter,
            IMyServiceGetter myServiceGetter)
        {
            this.myServiceSetter = myServiceSetter;
            this.myServiceGetter = myServiceGetter;
        }

        [FunctionName("Demo")]
        public void Demo([TimerTrigger("0 */5 * * * *", RunOnStartup = true)]TimerInfo timerInfo, ILogger log)
        {
            int value1 = 123;
            myServiceSetter.SetValue(value1);
            int value2 = myServiceGetter.GetValue();

            // Logs '123 - 0'
            // It means that MyServiceGetter and MyServiceSetter got different IMyservice instances.
            // Should be the same since we are using AddScoped to register services.
            // ---
            // If you comment out MyHttpClient injection from MyServiceGetter, it logs '123 - 123'.
            log.LogInformation($"{value1} - {value2}");
        }
    }
}

[t-l-k]

I believe this is the same as my issue #4914 - clearly something wrong with scoped registrations.

@fabiocav can we get an update on these tickets? Are they being investigated? Thx

[APIWT]

@fabiocav Hope you are having a nice day :) We are also hopeful that this issue will be fixed soon. We have spent a lot of time writing code to wire up our services and we really love the way scoped services work in ASP.NET Core. Let me know if there is anything I can do to help get this moving!

[buyckkdelaware]

We have experienced the same problem, but I also noticed this: It's only the first time after the function started/triggered, that the scoped services are 'wrong'. The second time, it seems to be fine. Has anyone else noticed this? (I was able to reproduce it with the code posted here by @heikkilamarko.)

This is a pretty big problem for us as well, any progress on this issue?

Some additional information: Azure Functions Core Tools - 2.7.1846 Function Runtime Version - 2.0.12858.0

[t-l-k]

@buyckkdelaware during load testing I have found multiple instances of same service type within the same request scope, it occurs frequently for me, at least 1 in 100 requests. I don't know if the weight of the type is relevant, but my DbContext subclasses are definitely affected - I get missed saves because entities get written to an incorrect DbContext, injected into either repositories, or other infrastructure code. I haven't yet managed to run more than 500 requests due to the DryIOC container scope exploding (which appears to be due to DefaultHttpClientFactory, also likely implicated in the HttpClient errors you've witnessed).

Detecting these duplicates is quite easy, using an AsyncLocal static collection in the affected class's constructor to track when more than 1 distinct hashcode has been found in. DbContexts are not thread safe ofc so can't be used by multiple threads anyway.

I haven't yet "handled" it, but I'll probably just throw an exception and fail the request thread using hashcodes as the detection basis. Without handling it, my code sometimes doesn't know that it has failed, it just misses data.

[darrenhull]

Are there any workarounds for this issue?

[fabiocav]

@darrenhull the workarounds will require some code refactoring at the moment. This is being assigned to a sprint for investigation, so we hope to have an update on this issue soon.

[darrenhull]

Hi @fabiocav,

Do you know if there is an update on this issue yet?

Sorry to keep asking but It’s a pretty big blocking issue for my company’s migration which we are pushing further and further away without being able to suggest when we can resume it.

Thanks

[darrenhull]

Hi @fabiocav,

I see this has been triaged and had the p2 tag. What exactly does the p2 mean, guessing priority2?

Can you give us an update please?

Thanks

[fabiocav]

@darrenhull yes, this is a priority label. The issue has been triaged, but not yet assigned. We'll have the milestone updated once this is assigned to a sprint.

[daniel-white]

@fabiocav: this really sounds like a .net core bug/issue. can you confirm?

i'm running into something similar using a vanilla asp.net core service.

[bartdk-be]

@fabiocav : Not to blame anyone, but how can a basic feature like Scoped or not being labeled as a p2 ? This seems like a rather basic functionality ? Can you point us to workarounds to resolve this issue ?

[daniel-white]

This appears to be a problem on at least .NET Core 2.1, but is fixed in 3.x. I've raised it here: dotnet/runtime#34586

[bartdk-be]

@daniel-white Any pointer on where it should be fixed in 3.1 ? Because we faced a similar issue in 3.1 (using the functions DI setup). Thanks !

[daniel-white]

@bartdk-be: im not sure - in my miminal example at dotnet/runtime#34586 make sure your Microsoft.Extensions.Http is the latest version.

[rodhemphill]

Hi @t-l-k ... I guess you don't have a work around then? Same significant problem here - fighting with it for months. It seems to happen on all but trivial systems - certainly with three of our serverside blazor systems with between 30 and 50 services. We are thinking of writing a state memory solution - creating a "StateMemoryService" as a singleton and having it register the instances of each other services against the transaction's SessionId. An in-memory register of <sessionId, serviceType, instanceId>. It's a LOT of effort for what is essentially a bug. @fabiocav @daniel-white

[melbappdev]

Hey @fabiocav, if you are up for fixing this I can help. I have three client projects that have the problem and I can easily produce a situation that doesn't get the problem. I can't give you client repos but I have a testing repo that I can share with you. The problem is easy to reproduce and demonstrate in that repo. This repo includes EF and various service classes, some of which cascade inject other services ... which are the most likely candidates for the problem. Whatever causes it the symptoms are that all "scoped" services get translated to "transient".

[t-l-k]

@rodhemphill sticky tape and plasters I'm afraid. Basically failing requests and relying on the runtime to execute the request again with fingers crossed for subsequent attempts. We detect the problem in constructors of classes we know should be scoped, and thus when two instances are created in a logical execution scope (determined by AsyncLocal) we throw an exception and log it. Makes a mess of my logs - see https://github.com/Azure/azure-functions-host/issues/4914#issuecomment-601251057.

It caused quite an embarrasing false start to our UAT phase for sure (deadlines and all), customer was quite alarmed at the amount of failing transactions, and that was as I was chasing it here to get it fixed. I had to refactor in the end, as is the advice per @fabiocav. even changing the order of parameters in constructors reduced the pain! It's still failing on some non-critical code paths, but we can live with that.

Agree- anything non-trivial causes it to fail. We built a polyglot solution, but one component was full DDD stack based on Microsoft's own patterns advice due to the domain complexity. Our simpler Functions Apps (just plain HTTP and read/write storage) work without issue.

We've dropped functions from our list of candidate dotnet technologies for future products & projects in favour of plain ASP.NET Core until this is fixed. Obviously we lose the serverless benefits.

Still, I managed to get money back from Azure for the development headaches it caused, as it is a GA product. If you're suffering the same I would recommend you to also raise support requests. Of course, it didn't come close to covering the amount of expense, inconvenience, and damage to our reputation. I had a very professional support experience with Microsoft and their Functions support team were amazing. But they couldn't fix the underlying issue - that's clearly an engineering problem.

Perhaps they might treat this more seriously if having to refund more customers via support requests ......... 😉

[espray]

I have had all of my teams stopped all .Net Core AzFunc development, this Scoped DI issue was the final straw for me. If basic .Net Core DI dose not work who knows what other problems may show up at the wrong time, the risk is too great.

May be send a message/tweet to Damian Edwards @DamianEdwards and/or Scott Hanselman @shanselman about your AzFunc development experience.

[AnthonyIacono]

I think the thing that bothers me about this issue is that it has been around in some form or another for like a year and a half. The original bug was a two line fix, and I imagine the issue we are all facing currently is similar. Can we please get an update from engineering?

[darrenhull]

We too have had to stop using @azfunc as the scoping issue makes them unusable for our solution. This is very frustrating for a team that have embraced Azure functions from day one. I’ve been very disappointed with how long this issue has taken to be resolved @fabiocav , oh wait it’s still active! We too are seeking money back on our subscriptions.

[melbappdev]

I have wasted about 4 days solid on this issue and with impending delivery dates and (soon to be) an unhappy client ... I am glad you responded @t-l-k as you've saved me a lot more wasted time. Yes I agree @darrenhull that it sounds like it has taken too long to fix, but a more important question is: what has it cost? How many people like me have fallen into this hole? @fabiocav

[Arash-Sabet]

This issue is a huge blocker and addressing it has to be expedited. If there's a P1 label indicating a higher priority than P2, that one should be applied instead of P2. We cannot proceed with our project gracefully without a fix! @fabiocav

[fabiocav]

Assigning this to the next sprint. @Arash-Sabet, if you can open an issue detailing your scenario, we might be able to assist with a workaround until this is resolved.

[fabiocav]

The original bug was a two line fix, and I imagine the issue we are all facing currently is similar.

@anthonychu thank you for the patience, can you please share more details on what fix you're referring to?

[Arash-Sabet]

@fabiocav thanks for your reply. I will try to get you a bare-minimum project tomorrow. The only workaround that I could think of was to avoid the "scoped object" dependency injection and instead pass the "scoped" object's instance by adding an interface property to the consumer/service object. I was just wondering, what workaround you may be thinking of so that I could think about it ahead of time? Thanks.

[vitalybibikov]

Impossible to implement Domain Events with Azure functions without this functionality properly... Can't believe it does not work. Still....

I'm asking to change the priority, as this is a blocker.

Thanks

[vitalybibikov]

Hi @fabiocav Any news on this?

Can I somehow assist to speed this one up?

Thanks

[vitalybibikov]

As my solution is heavily using DDD, it was not possible for me to ignore absence of this core functionality.

I am using Bindings a lot, so moving to App Service is also not an option at the moment. For anybody who is planning to use AF in prod, I'm strongly advising not to do so, as they are definitely not in Production Ready state for anything except for "web hooks".

The solution for now is:

To emulate scoped by using AddSingleton and AddTransient, so it will be possible to store all the connections in a Singleton dictionary and grab them from their when necessary.

Connection Key: FullTypeName + Request HashCode (or anything you want)

In startup it looks like this:

services.AddScopedContext<ITenantDbContext, TenantDbContext>();

Some boilerplate is required in order to dispose everything on exit. My solution is to use DisposeContextFilter class that encapsulates all the disposal logic.

As filters in AF are not in a ready-to-use state also, it looks ugly. Though it works. I can share the code if necessary.

[Boaz101]

I am also experiencing issues with an Azure function which uses scoped services combined with AddHttpClient. If I comment the AddHttpClient from startup it works correctly. I am moving the HttpClient out of dependency injection, however that has potential pitfalls, too. Hopefully, this is fixed soon because this is causing us a lot of headaches.

[roszell]

@fabiocav any news on this? I see it was assigned to Sprint 75 that was marked closed, but this was the only issue not resolved as part of that. Is this actively being worked on still?

[Schlebusch]

Sooooo ... any news? Glad to find this issue early enough to stop using scoped dependencies in my Azure Functions project. It is frustrating to read that this issue is open for 9 months now and nothing indicates that it will be solved at all ... MSFT should create a list of features in their SDK's that are not working.

[APIWT]

@fabiocav Hope you are having a nice day :) Any updates on this?

[t-l-k]

@fabiocav please can you advise as to any updates for this issue? We've some gaps opened up in our roadmap in August in which we're looking forward to paying down some technical debt, and revisiting this issue is on our tracker alongside migrating to dotnet 3+.

Is this recognised as a potential https://github.com/dadhi/DryIoc issue? Is there a correlating ticket with the DryIOC project?

Thanks

[espray]

@jeffhollan @brettsam any update on this issue?

[alexanderbikk]

hi guys I'm using .net core 3.1 and looks like Scoped services work well but not for the "cold start". So, when I run the function app either on local or Azure env and the function does the "cold start", my scoped services will be injected more than once per request. When I do the requests after "cold start" seems like all good.

So, for example EF DbContext will be created more that once per request for the "cold start". But for all next requests it will be only the one instance per request. I'm wondering does it the same issue or I'm doing something wrong?

btw same for my custom services. I create the UoW which includes the EF DbContext, and registered it as Scoped. And again got the same issue.

Thanks

[rodhemphill]

Hi @alexanderbikk. One thing to check is whether you have a "prerender" process running that creates the initial instance. This is a known problem with net core Blazor apps for example.

[alexanderbikk]

Hi @rodhemphill I'm not sure that I get you.

As I said the Scoped lifitime works sometimes, but sometimes I get different instances per request. I can reproduce it on "cold start". Then everything works fine. After some idle the function does a "cold start" again and I again get this issue.

Thanks

[brettsam]

@alexanderbikk -- would you be able to throw a simplified repro up somewhere (in a github repo or even just a gist)? That'd help get this set up...

[alexanderbikk]

hi @brettsam

I believe I have not so much time for it. But I can describe the details and provide some code. You can try the following scenario:

• Create the Azure function v3 with latest sdk version.
• Create two or more "Services" and UoW which can be configured as Transient in function Startup
• Add EF core DbContext which will be configures as Scoped by default. I used separate project with EF Core and just configuring the DbContext in function Startup.
• Use created DbContext in "Services" and UoW (DI in constructors)
• Call UoW and "Services" in az function to check the injected DbContext instance
• Also you can add some kind of logic in Services and using UoW to save changes. But it will be enough to check the injected DbContext InstanceId

The expected behavior is that the DbContext instance should be the same per one function call But in my case sometimes(I reproduced it on "cold start") the "Context" instance is different for different Services, so I got some kind of Transient behavior.

Here is possible example of Startup.cs and Services

public class Startup : FunctionsStartup
{
    public IConfigurationBuilder ConfigurationBuilder { get; }

    public override void Configure(IFunctionsHostBuilder builder)
    {
        builder.Services.AddLogging();

        builder.Services.AddHttpClient();

        builder
            .Services
            .AddOptions<ApplicationSettings>()
            .Configure<IConfiguration>((settings, configuration) => { configuration.Bind(settings); });

        ApplicationSettings config = builder.Services.BuildServiceProvider().GetRequiredService<IOptions<ApplicationSettings>>().Value;
        builder.Services.AddDbContext<DatabaseContext>(options => options.UseSqlServer(config.WebSql.ConnectionString));

        builder.Services.AddTransient<IService1, Service1>();
        builder.Services.AddTransient<IService2, Service2>();
        builder.Services.AddTransient<IUnitOfWork, UnitOfWork>();

    }
}

public class Service1 : IService1
{
    private readonly DatabaseContext dbContext;

    public Service1(DatabaseContext dbContext)
    {
        this.dbContext = dbContext;
    }
}

public interface IService1
{
}

public class Service2 : IService2
{
    private readonly DatabaseContext dbContext;

    public Service2(DatabaseContext dbContext)
    {
        this.dbContext = dbContext;
    }
}

public interface IService2
{
}

public class UnitOfWork : IUnitOfWork
{
    public DatabaseContext DbContext { get; }

    public UnitOfWork(DatabaseContext dbContext)
    {
        DbContext = dbContext;
    }

    public void BeginTransaction()
    {
        DbContext.Database.BeginTransaction();
    }

    public async Task BeginTransactionAsync()
    {
        await DbContext.Database.BeginTransactionAsync();
    }

    public void Commit()
    {
        DbContext.Database.CommitTransaction();
    }

    public void Rollback()
    {
        if (DbContext.Database.CurrentTransaction != null)
        {
            DbContext.Database.RollbackTransaction();
        }
    }

    public void SaveChanges()
    {
        DbContext.SaveChanges();
    }

} 

UPD: And one important note that I'm using bot UoW and Services not directly in function but inside the other class, let's call it Manager. Which injected and called from azure function.

public class Manager : IManager 
{
    private readonly IService1 _service1;
    private readonly IUnitOfWork _unitOfWork;

    public StateMachineActionsInvoker(IService1 service1,
        IUnitOfWork unitOfWork)
    {
        _unitOfWork = unitOfWork;         
        _service1 = service1;
    }
    public void Call()
    {           
       //do something         
     }
}

public class TestFunction
{
    private readonly IManager _manager;

    public TestFunction(IManager manager)
    {
        _manager= manager;
    }

    [FunctionName("TestFunction")]
    public async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest request,
        ILogger log)
    {
        _manager.Call()
    }
}

UPD2: @brettsam I have tried to reproduce it and no luck

I think I found the issue. At least I found the case when it happens. I called the dbContext.SaveChanges() inside my services and outside of services I called UoW, to use the transaction. In this case the injected dbContext (InstanceId) was different for UoW and my Service. But again it was only on "cold start" really strange behavior.

I have refactored my code, and called dbContext.SaveChanges() outside of the services from my UoW and I can't reproduce the issue. Seems like all good and nothing to check

Thanks a lot.

[pragnagopa]

@brettsam - Any updates here? Should we move this next sprint?

[brettsam]

I haven't been able to investigate yet -- but I've moved it to the next sprint so it comes back on @fabiocav's radar.

[fsalmeida]

I've been facing the same problem and I've found a workaround. While I was testing some scenarios, I found out that the problem only happens on classes that expect the HttpClient as DependencyInjection. For example:

` public class SomeClass { private readonly MyHttpClient myHttpClient; private readonly MyScopedDependency myScopedDependency;

public SomeClass(MyHttpClient myHttpClient, MyScopedDependency myScopedDependency)
{
    this.myHttpClient = myHttpClient;
    this.myScopedDependency = myScopedDependency;
}

public void SomeMethod()
{
    myHttpClient.DoSomething();
}

} `

In this scenario, the "MyScopedDependency" would not be resolved as Scoped, but if I change my code to something like this:

` public class ClassToHoldHttpClient { public MyHttpClient MyHttpClient { get; }

public ClassToHoldHttpClient(MyHttpClient myHttpClient) => MyHttpClient = myHttpClient;

}

public class SomeClass { private readonly ClassToHoldHttpClient classToHoldHttpClient; private readonly MyScopedDependency myScopedDependency;

public SomeClass(ClassToHoldHttpClient classToHoldHttpClient, MyScopedDependency myScopedDependency)
{
    this.classToHoldHttpClient = classToHoldHttpClient;
    this.myScopedDependency = myScopedDependency;
}

public void SomeMethod()
{
    classToHoldHttpClient.MyHttpClient.DoSomething();
}

} `

With this code, "MyScopedDependency" is now beeing resolved as Scoped again.

Is it a great solution? Of course not, but as this problem is not solved for about to 1 year, maybe this workaround may save some people.

[GuhBigardi]

I've been facing the same problem and I've found a workaround. While I was testing some scenarios, I found out that the problem only happens on classes that expect the HttpClient as DependencyInjection. For example:

` public class SomeClass { private readonly MyHttpClient myHttpClient; private readonly MyScopedDependency myScopedDependency;

public SomeClass(MyHttpClient myHttpClient, MyScopedDependency myScopedDependency)
{
    this.myHttpClient = myHttpClient;
    this.myScopedDependency = myScopedDependency;
}

public void SomeMethod()
{
    myHttpClient.DoSomething();
}

} `

In this scenario, the "MyScopedDependency" would not be resolved as Scoped, but if I change my code to something like this:

` public class ClassToHoldHttpClient { public MyHttpClient MyHttpClient { get; }

public ClassToHoldHttpClient(MyHttpClient myHttpClient) => MyHttpClient = myHttpClient;

}

public class SomeClass { private readonly ClassToHoldHttpClient classToHoldHttpClient; private readonly MyScopedDependency myScopedDependency;

public SomeClass(ClassToHoldHttpClient classToHoldHttpClient, MyScopedDependency myScopedDependency)
{
    this.classToHoldHttpClient = classToHoldHttpClient;
    this.myScopedDependency = myScopedDependency;
}

public void SomeMethod()
{
    classToHoldHttpClient.MyHttpClient.DoSomething();
}

} `

With this code, "MyScopedDependency" is now beeing resolved as Scoped again.

Is it a great solution? Of course not, but as this problem is not solved for about to 1 year, maybe this workaround may save some people.

\o/ It worked for me, thank u @fsalmeida, but I hope that fix coming soon :)