Problems with scoped lifetime services (AddScoped)

[heikkilamarko]

Hi,

We are using durable functions with dependency injection. In our Startup.cs file, we register our dependencies as follows:

services.AddScoped<IMyService>(serviceProvider => new MyService(...))

In our activity function, we are using normal constructor injection to get IMyService instances.

The problem is that even if we are using AddScoped for registering the service, during the activity function run, each class that asks for the service, gets a different instance of IMyService. This breaks our app logic, because IMyService users won't see each other's changes.

As a workaround to the earlier Azure Functions runtime DI issues, we had the following pinning in place FUNCTIONS_EXTENSION_VERSION = 2.0.12673.0. The pinned version is not supported anymore by Azure, so our function DI is now broken again.

[espray]

@brettsam @fabiocav @jeffhollan @shanselman @DamianEdwards this was part of Spring 84, but was not completed. Now its not part of Sprint 85! WTF! Its coming up on 1yr for this P2 issue. Not building a lot of confidence in the community this way.

[vitalybibikov]

@espray I guess that's some kind of development, when you can just drop things out of a scope forever and ever. As I can see 2 or 3 things in each of their sprints, that are not closed and just hang there without being tackled.

I've been waiting almost for a year for this thing to be closed. But I guess something like Scoped is not a priority, so I've just created my own Scoped extension out of Singleton and Transient for one of my customers.

Now, as an architect in my organization I'm advising client's not to use Azure Functions, as they are still not ready for production use.

[espray]

@vitalybibikov same situation here. Actively not recommend AzFunc as well. Mainly because of HOW this issue has been handled.

[fabiocav]

Apologies for the delay on this. We have been, we have a relatively small team, and principally during the current circumstances, it has been challenging to address some of the issues in our sprints, and this was unfortunately, one of them.

I've been working with @brettsam and @anthonychu to resume the work on this. I also apologize for the lack of communication here. We'll keep this issue updated as we move forward.

Thank you for the patience.,

[fabiocav]

Providing an update here at the end of the week. @brettsam and I have made some progress on the different scenarios reported, testing and investigation, hopefully we'll have something ready before the sprint is over, but will continue to provide updates here for awareness.

[APIWT]

@brettsam Glad to see the fix was very simple. Would you be able to provide a timeline on deployment?

[brettsam]

The code change was small, but changes like this can be pretty impactful, which is why we decided to hide it behind a feature flag and get that change out ASAP. We wanted to let everyone try it and let us know if they run into issues while we continue testing scenarios. If all goes smoothly, we will make this the default everywhere in a future release.

As for the timeline -- it's currently in validation and the deployment is starting this week. We don't like giving exact timetables (hotfixes, unrelated incidents, etc can delay a deployment)... but it generally takes 6-ish business days to roll out worldwide once it has started.

[APIWT]

@brettsam Sounds good! Is there an easy way to be notified when the runtime on our functions app resource has support for the feature flag?

[darrenhull]

Thanks for fixing the issue and for the feedback @brettsam and @fabiocav. I too would like to know when it is available.

[t-l-k]

A quick GitHub search: https://github.com/Azure/azure-functions-host/wiki/Configuration-Settings -

AzureWebJobsFeatureFlags: Set to a comma delimited list of beta features to enable. Beta features enabled by these flags are not production ready, but can be enabled for experimental use before they go live.

I'm putting this in the AppSettings of my test deployments posthaste and grabbing the popcorn:

AzureWebJobsFeatureFlags = EnableEnhancedScopes

Meanwhile I'll be watching https://github.com/Azure/azure-functions-host/releases and https://github.com/Azure/azure-functions-core-tools/releases like a hawk waiting for this to drop.

[fabiocav]

The validation phase has been completed and the release is starting to rollout globally. We don't usually publish the Core Tools until after the deployment, but we're making some changes to the process and will see if we can get this up ASAP so you can test locally as well.

Again, thanks for the patience with this!

[fabiocav]

We'll actually keep this issue open (the original intention was to keep it open) so we can continue to use this for feedback/validation.

[ebwinters]

Guys I have the solution after hours of investigation: Instead of async void for function return type, do async Task. Not sure why this fixes it, any explanation would be great. @fabiocav

[jeffhollan]

In C# you should only use async void when you have to (event handlers, which functions are not in that sense), so likely due to that https://stackoverflow.com/questions/13636648/wait-for-a-void-async-method

[ishepherd]

Not sure why this fixes it, any explanation would be great.

@ebwinters your function would return when it hits the first await. I suppose the functions host has no way of telling there is still work pending, because you did not return a Task for it to wait on.

The async void case is a “fire and forget”: You start the task chain, but you don’t care about when it’s finished. When the function returns, all you know is that everything up to the first await has executed. Everything after the first await will run at some unspecified point in the future that you have no access to. https://devblogs.microsoft.com/oldnewthing/20170720-00/?p=96655

[ebwinters]

Good to know

[t-l-k]

This is now working great for me, my failed request rates and exception rates have basically dropped to zero, and applications are working better than ever. This brings Azure Functions back to the table for projects in flight!!!

p.s. I skipped local validation using the func.exe tools and just went for the docker image mcr.microsoft.com/azure-functions/dotnet:3.0.14785, which worked great as well.

[brettsam]

@t-l-k -- to be clear, you're running with the new feature flag?

[t-l-k]

@brettsam indeed! Unless my eyes deceive me!

[fabiocav]

@t-l-k great! Thanks for providing feedback on this! And again, thank you for the patience with this! If you find any other issues while enabling this toggle, please let us know as we'll be making this the default once we have some good validation.

For others who were experiencing the same issue, please provide feedback if possible.

Guys I have the solution after hours of investigation: Instead of async void for function return type, do async Task. Not sure why this fixes it, any explanation would be great.

@ebwinters the information provided by @ishepherd is correct, the pattern you were using will trigger the behavior you're describing and other potential side effects as well (e.g. early termination of your function if the host is deallocated). Without a Task that represents the work being executed, the calling code doesn't have any information about when that work completes and the invocation will be completed as soon as the method returns. By the way, this is not unique to Azure Functions, but applies to .NET in general. We do have checks and warnings in the runtime (if you were to look at you app diagnostics, you'd see a warning about async void usage) to make you aware of that.

[ebwinters]

@fabiocav that makes sense, thanks for explaining. I saw some code snippets in this thread with a void return type, so this may still help them (assuming warnings are being suppressed)

[ghost]

@fabiocav Applying the feature flag helped in our case as well.

Our story: Last month, we noticed some odd behavior regarding scoped services. In our case, a scoped service was responsible for setting up one database connection per request if necessary, and disposing it once the scoped service was disposed. However, after a longer period of time without restarts, the function app ran out of ports. After some (long) investigation we realized that scoped services were sometimes not disposed correctly. Then I came across this issue 2 weeks ago and we applied the feature flag to the function app that showed this behavior. After that we let it run for a while, and it seems that we no longer leak connections, which suggests that the scoped service is being disposed of properly now.

After 2 months since the last response, is there any idea of when this will be available by default without having to add the feature flag?

[brettsam]

Thanks for providing that data, @b10-dslappendel! This is very useful.

We'll discuss this and see if we have enough usage and data to make it the default.

[simongullberg]

I can tell that adding "AzureWebJobsFeatureFlags": "EnableEnhancedScopes" solved the problem for us when injecting a scoped service.

We were injecting a scoped EF DbContext and this context was injected in two transient services (let's call those Service1 and Service2). Service1 fetched some entity from the database with the injected DbContext and then sent the entity to Service2 where the entity was modified and a call to SaveChangesAsync() was made on the injected DbContext in Service2. The update did not work because there was no change when looking in the database. I suspected that something was wrong with the DbContext so I added an instance variable of type Guid and called Guid.NewGuid. I then expected to see the same Guid on the DbContext injected in Service1 and Service2 but they were different. After adding the feature flag EnableEnhancedScopes the Guids were the same in both Service1 and Service2 and the entity was updated in the database.

[danielreuterwall]

We've had similar issues. Change tracking didn't seem to work between EF DbContext injected directly in function code and EF DbContext injected in scoped services. Using the feature flag did solve that problem for us which is great.

However, in a service added using AddHttpClient in our Startup.cs we injected IHostEnvironment environment to determine if we were in a production environment or not. After the feature flag was set IHostEnvironment environment was no longer available. We've solved it using a regular configuration variable, but the reason for why it's no longer available is unclear. We also tried to revert by removing the feature flag configuration but is seems like once it's set it's there for good.

So two questions: Any reason or explanation why IHostEnvironment environment is not longer available? Is it possible to remove the feature flag?

[VaclavK]

stumbled upon this while investigating issue with incomplete ef core (3) persistance..

our pattern is that we have multiple repos supposedly using (scoped) DbContext and we call save changes on one of them - this generally works (is meant to) but in one function app we have http client injected into DI

we started seeing comply partial data persistence (from the repo that was saved, not from the others) which this issue explains

this feature flag - is this PROD ready - assuming this is considered beta flag so it would not be recommended for production usage??? @fabiocav

reproducible in azure (v3 functions) and locally with azure functions CLI - 3.23.0

interestingly enough - we were trying to understand why we would not see this issue consistently/with high enough frequency - our (affected) function is triggered in parallel by SB trigger and behaviourally it seems not to be reproducible with message pump invoking function in parallel (in default config up to 16 messages)

[t-l-k]

@VaclavK I wouldn't consider Azure Functions "production ready" when running without this flag TBH, especially if you're running a design with a properly encapsulated domain model with plenty of service abstractions.

@brettsam Is it enabled by default yet?

[fabiocav]

@t-l-k I'm sending the changes to make this the default in the current sprint. Next release will have the new behavior.

[VaclavK]

@fabiocaFabie just so I understand this - This will be the default behaviour without need to configure anything in?

How do I know that when that release will be the release running in azure for my functions?

[swettstein]

@fabiocav there was a release 2 days ago (3.0.15733). can you confirm this was part of that release?

[fabiocav]

@swettstein are you referring to the change in defaults? If so, the change was not made because of concerns with a few other items landing right before //build and some announcements.

My goal is to have the defaults change with the following release.

[mintyfusion]

We ran into the same issue, but adding AzureWebJobsFeatureFlags = EnableEnhancedScopes is making things worse, where all Scoped objects behave as Transient.

We are using Azure Functions 3.x and I have tried both Windows and Linux and result is exactly the same.

[fabiocav]

@mintyfusion can you please open an issue with the details of what you're seeing a a repro? What you're describing above leads me to believe something is incorrect with the setup and it would be good to focus on that.

[swettstein]

@fabiocav any update on releasing "EnableEnhancedScopes" so we don't need the feature flag anymore?

[Socolin]

Hello,

~I also have this issue, but EnableEnhancedScopes dont fix the problem, is this supposed to works with the new azure functions (.net 5 + runtime: dotnet-isolated)~

Edit: I finally found the problem, there was a service registerd as Singleton and depending on a Scoped dependency (I was expecting an exception instead of a silent new())

[qcnguyen]

any updates on this ? team ? @fabiocav

[swettstein]

@fabiocav any update please?

[Schuer84]

Any updates on this one? We registered our DbContext as scoped, but we noticed, if we start multiple http requests, the context gets shared between the different requests, which leads to threading issues:

{ type: 'NotSupportedException', message: "A second operation started on this context before a previous asynchronous operation completed. Use 'await' to ensure that any asynchronous operations have completed before calling another method on this context. Any instance members are not guaranteed to be thread safe." }

Any feedback on how to fix this would be greatly appreciated. PS: We enabled the flag "AzureWebJobsFeatureFlags": "EnableEnhancedScopes",

[fabiocav]

@Schuer84 can you please open a separate issue with those details so we can take a closer look at your scenario? We have no evidence that with the changes made behind the flag still have issues with scoped services, so it would be good to rule other things out.

Thanks!

[Gevil]

Alright some new info on this in our functions we are also using HttpClients with DI with AddHttpClient.

After enabling EnableEnhancedScopes we are seeing fewer errors related to object disposed exceptions from DryIoc related to our DbContexts when locally debugging.

As soon as we deploy our Function to Azure and run tests on it everything falls apart and almost every function trigger fails.

{ "Error": 39, "StackTrace": " at DryIoc.Throw.It(Int32 error, Object arg0, Object arg1, Object arg2, Object arg3) in D:\\a\\1\\s\\src\\WebJobs.Script.WebHost\\DependencyInjection\\DryIoc\\Container.cs:line 9000\r\n at DryIoc.Container.ThrowIfContainerDisposed() in D:\\a\\1\\s\\src\\WebJobs.Script.WebHost\\DependencyInjection\\DryIoc\\Container.cs:line 412\r\n at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in D:\\a\\1\\s\\src\\WebJobs.Script.WebHost\\DependencyInjection\\DryIoc\\Container.cs:line 201\r\n at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in D:\\a\\1\\s\\src\\WebJobs.Script.WebHost\\DependencyInjection\\DryIoc\\Container.cs:line 196\r\n at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedServiceProvider.GetService(Type serviceType) in D:\\a\\1\\s\\src\\WebJobs.Script.WebHost\\DependencyInjection\\ScopedServiceProvider.cs:line 25\r\n at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)\r\n at Microsoft.EntityFrameworkCore.Internal.ScopedLoggerFactory.Create(IServiceProvider internalServiceProvider, IDbContextOptions contextOptions)\r\n at Microsoft.EntityFrameworkCore.Infrastructure.EntityFrameworkServicesBuilder.<>c.<TryAddCoreServices>b__7_0(IServiceProvider p)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)\r\n at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)\r\n at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)\r\n at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)\r\n at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()\r\n at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider()\r\n at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()\r\n at Microsoft.EntityFrameworkCore.DbContext.get_Model()\r\n at Microsoft.EntityFrameworkCore.Internal.InternalDbSet1.get_EntityType()\r\n at Microsoft.EntityFrameworkCore.Internal.InternalDbSet1.get_EntityQueryable()\r\n at Microsoft.EntityFrameworkCore.Internal.InternalDbSet1.System.Linq.IQueryable.get_Provider()\r\n at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.Include[TEntity,TProperty](IQueryable1 source, Expression1 navigationPropertyPath)\r\n at Gsop.Cpo.ChargeLocationService.Business.Services.OpeningTimesService.GetRegularOpeningHoursPeriodsAsync(String chargeStationId, DateTime utcNow) in D:\a\1\s\Gsop.Cpo.ChargeLocationService.Business\Services\OpeningTimesService.cs:line 348\r\n at Gsop.Cpo.ChargeLocationService.Business.Services.OpeningTimesService.GetAllPeriodsForAllChargeStations(String chargeStationId, DateTime utcNow) in D:\a\1\s\Gsop.Cpo.ChargeLocationService.Business\Services\OpeningTimesService.cs:line 182\r\n at Gsop.Cpo.ChargeLocationService.Business.Services.OpeningTimesService.ProcessOpeningHoursPeriods(String chargeStationId, Nullable1 utcNowOverride) in D:\\a\\1\\s\\Gsop.Cpo.ChargeLocationService.Business\\Services\\OpeningTimesService.cs:line 35\r\n at Gsop.Cpo.ChargeLocationService.Functions.FunctionsTimerTriggers.CalculateOpeningTimes(TimerInfo myTimer) in D:\\a\\1\\s\\Gsop.Cpo.ChargeLocationService.Functions\\Functions.Timer.cs:line 36", "Message": "Container is disposed and should not be used: Container is disposed.\r\nYou may include Dispose stack-trace into the message via:\r\ncontainer.With(rules => rules.WithCaptureContainerDisposeStackTrace())", "Data": {}, "Source": "Microsoft.Azure.WebJobs.Script.WebHost", "HResult": -2146233079 }

The function Version is V3 we're running .net core 3.1 and ef core 3.1 version.

This is ridiculous that we can't use proper DI and HttpClient injection with functions runtime and packages that are in production.

You need to provide at least a workaround otherwise there's no point in using azure functions whatsoever.

Without the Feature flag we can at least get our functions running and only encountering the issue rarely.

[thomasvdb]

@Gevil The workaround we found when using HttpClients is by adding an extra abstraction layer and don't inject an HttpClient (or HttpClientFactory) directly. This is now running for about 1 year in production without any issues.

Example:

public class FunctionWorkaround
{
        private readonly IHttpClientFactory httpClientFactory;

        public FunctionWorkAround(IHttpClientFactory httpClientFactory)
        {
            this.httpClientFactory = httpClientFactory;
        }

        public HttpClient CreateClient() => httpClientFactory.CreateClient(...);
}

[eluchsinger]

@thomasvdb so what do you inject into the Function then? In the example you are injecting an IHttpClientFactory. Do you inject the FunctionWorkaround?

[eluchsinger]

@fabiocav @brettsam This issue looks like related to my new issue reported. I don't think that it's the same issue though so I wouldn't merge them.

https://github.com/Azure/azure-functions-dotnet-worker/issues/628

[VaclavK]

Our strategy so far dealing with this

in our case (given this is your vanilla ef.core n-tier app with db store, this may be applicable for other folks) was that due to separate db context (and its associated changed entities), data was not being saved as save changes was invoked on the primary db context ,not the secondary

we have moved all functions that talk to Http (fortunately that is few integration points in our apps) to a separate fn app... these are woken by commands and do the http dance and persist whatever they retrieve etc into db store, for many of the data services we have added an optional param to send a repository object to interact with DB - so as to skip the one injected by DI (as that is tied to a separate db context due to this bug)

this way we gain additional runtime isolation - not to frown upon - all tied to the same fn app plan so the overhead is not huge - and the code change associated is easy to dissolve once this is resolved (and we could have done feature flag)

[thomasvdb]

@eluchsinger Indeed, I inject the FunctionWorkaround (it has an interface as well in our case, left it out in the example).

[swettstein]

@swettstein are you referring to the change in defaults? If so, the change was not made because of concerns with a few other items landing right before //build and some announcements.

My goal is to have the defaults change with the following release.

@fabiocav any update on making this feature flag default? i haven't seen/heard anything. our team has been tracking this issue for a long time and we have to add this feature flag to each app we make so would be nice that it gets moved to default.

[swettstein]

@fabiocav any update on making this feature flag default? i haven't seen/heard anything. our team has been tracking this issue for a long time and we have to add this feature flag to each app we make so would be nice that it gets moved to default.

in case anyone else is looking for the answer, i found it here: https://github.com/Azure/azure-functions-host/pull/7669

seems that it will only be removed in v4, not v3

[soloham]

I've just experienced this issue when using middlewares (.NET 6, isolated), I have got an auditing scope defined in the middleware before the function call where I resolve a (scoped) EF DB context here and would like to call SaveChanges on this DB context at the end of the request pipeline, but it seems this DB context is different to the one that is resolved in the function, so doing save changes at the end does nothing as the changes never got tracked on this DB context.

Please let me know if this is a bug or expected behavior for the host and if expected how can I achieve the above goal. Thanks!

@fabiocav @brettsam FYI

[brettsam]

@SoloHam -- can you open an issue in the https://github.com/Azure/azure-functions-dotnet-worker/issues repo? Please include some example code to show us how things are set up in your scenario.

This issue is related to the in-proc DI setup, which is different from that in the isolated worker.

[soloham]

It appears my issue is already answered (#811) in the appropriate repo. Thanks @brettsam!