No way to disable functions locally in isolated mode based on different environments (only using local.settings.json)

[Schmaga]

Description

We are currently migrating a .NET 6 in-process function app to isolated mode, and are running into issues when trying to disable functions during local development. Our requirements are to disable certain functions in some local environment settings, but enable them for others. Thus, simply putting stuff into the local.settings.json is no solution for us.

With the old in-process libs, we were using [Disable("Setting")] Attributes on the functions we wanted disabled, and we were configuring those settings in appsettings.<Environment>.json files we added during function startup using the regular AddJsonFile configuration extensions. The [Disable] Attribute does not work any more (at least currently not, related #312 and #438). And using the AzureWebJobs.<FunctionName>.Disabled approach only works, when putting those settings into the local.settings.json. It seems that somehow, all Functions are already loaded and configured before any HostBuilder user code executes. That means, adding any configuration in ConfigureHostConfiguration or ConfigureAppConfiguration is too late and will not disable any functions. We are not sure if this is by design, but it seriously constraints any dynamic configuration of functions when developing locally.

Expected behavior

We would expect to have any way to disable functions during startup with a more dynamic mechanism except the local.settings.json, depending on the configured environment.

Actual behavior

There is no way to locally disable functions by configuration except local.settings.json, at least none we found.

Known workarounds

We could use compiler flags and combine different environment with those compiler flags, to simply remove the affected functions from the code. But this feels extremely clunky, ugly and error-prone.

App Details

• Functions versions: Microsoft.Azure.Functions.Worker 1.10.0 Microsoft.Azure.Functions.Worker.Sdk 1.7.0 Microsoft.Azure.Functions.Extensions.Http 3.0.13 Microsoft.Azure.Functions.Extensions.Timer 4.1.0
• Azure Functions runtime version: v4
• Programming language used: C#

[fabiocav]

You're correct that this is a limitation with the existing model, but with the metadata generation updates, where the worker is responsible for the function metadata, this will be supported.

Flagging this for validation with the source generation model.

High level tasks associated with this:

• [ ] Define the disabled attribute on the worker, with support for conditions based on configuration
• [ ] Ensure the metadata generator is using the attribute and evaluating conditions
• [ ] Ensure the host is properly handling the disabled flag sent with the function metadata.

[Zoe1808]

You're correct that this is a limitation with the existing model, but with the metadata generation updates, where the worker is responsible for the function metadata, this will be supported.

Flagging this for validation with the source generation model.

High level tasks associated with this:

• [ ] Define the disabled attribute on the worker, with support for conditions based on configuration
• [ ] Ensure the metadata generator is using the attribute and evaluating conditions
• [ ] Ensure the host is properly handling the disabled flag sent with the function metadata.

@fabiocav Do we have ETA for this? We really need this 'Disable' functionality before we migrate to isolated mode.

[fabiocav]

Thank you for the follow up, @Zoe1808 . We don't have an ETA to share as this issue has not been assigned due to competing priorities.

@satvu let's follow up on this so we can scope the work, as this could be done with enhancements to the generator.

[TomHutsonVantage]

Are there any recent updates to this issue? My team is facing the same problem and the workarounds aren't very tenable.

[campbellja]

@fabiocav Thank you for the progress on this issue. My team shares the same problem as @TomHutsonVantage and we would like a maintainable way to disable timer triggers per environment or altogether.

[Rfaering]

We have the same problem

[dsossaman]

Our team has the same issue.

[davidpetric]

Have the same issue.

[Matthewsre]

Was revisiting isolated functions now that .net 8 is coming out (isolated was supposed to be the only/primary path in the roadmap with .net 8) and I believe this issue is one of the last remaining issues, now that we finally have better support handling ServiceBus messages and the Application Insights story has improved.

@fabiocav , any updates or traction on this issue?

[wc-whiteheadd]

This is also the thing which is restricting us to moving to the latest version.

[fabiocav]

Providing an update here as after evaluating the work, we're inclined to keep it consistent with the other stacks and support disablement using the same environment variables/App Settings (AzureWebJobs.<FunctionName>.Disabled).

@Schmaga I'm assuming you were injecting your configuration with a startup in the in-proc model, but that may lead to issues depending on the hosting model you're using (can you confirm what plan you use for your apps).

Additional folks on the issue, to summarize, if we go down this path, the following would be required to disable functions:

• For hosted scenarios, an app setting following: AzureWebJobs.<FunctionName>.Disabled would be used
• For local development, app settings are defined as environment variables in either local.settings.json or launch configuration.

Sharing the plans here to continue the conversation and get more feedback from you all and collect information about use cases that would be too challenging with what I've described above.

[tbasallo]

@fabiocav what about the disabled attribute that currently exists in non-isolated. See #9693 for more details on why this is different than using local settings.

Basically, though, local settings is not part of source control and would require some other way to synchronize for new devs or cloned projects. Much less new functions that are added and then kick off when the repo is sync'd but the settings are outdated.

The use case being, I almost never want my local environment to process every queue and timer trigger. I have functions apps with 30 and 40 functions on these triggers. It's much easier to

IF DEBUG

[DISABLE()]

Please do not remove this from isolated.

[dsossaman]

@fabiocav I agree with @tbasallo. The use of #IF DEBUG is straightforward in the dev environment and does not require adjusting local settings if you decide to change a function name, etc.

[SeanFeldman]

This is a very common pattern when running locally.

Similar to Timer triggered functions with RunAtStartup surrounded by #if DEBUG pre-processor directive to ensure a timer is executed upon execution.

[Schmaga]

@Schmaga I'm assuming you were injecting your configuration with a startup in the in-proc model, but that may lead to issues depending on the hosting model you're using (can you confirm what plan you use for your apps).

We are currently using .NET 7 Isolated mode functions. And yes, we are injecting those configuration settings with a Startup. Either using AddJsonFile in local environments, or via the AppConfiguration integration in Azure hosting scenarios.

Additional folks on the issue, to summarize, if we go down this path, the following would be required to disable functions:

• For hosted scenarios, an app setting following: AzureWebJobs.<FunctionName>.Disabled would be used
• For local development, app settings are defined as environment variables in either local.settings.json or launch configuration.

Would this mean that the approach using settings injected via the AppConfiguration provider would not work, instead we would always have to add them to the app settings of the App Service?

[fabiocav]

Would this mean that the approach using settings injected via the AppConfiguration provider would not work, instead we would always have to add them to the app settings of the App Service?

If we continue with the plans mentioned above, yes, that would be the case.

I am curious about your current in-proc setup, though. What hosting model are you running on?

We had a review of this issue today, and we do believe the root of the problem here is really around better configuration (as opposed to just supporting the disabled attribute model), and more significant configuration changes are required to address a few issues, including the local development flow.

[wc-whiteheadd]

Sorry for the delayed response to this, i have only just come back from holiday.

@fabiocav, I don't think that solution really fixes the underlining problem which has been introduced from the isolated worker model.

I say this because the entire reason for why people are using the appsettings and disabled attribute to enable/disable azure function triggers, is it to seperate concerns between infrastructure and development. In the above solution, for the isolated worker model, your still requiring either developers to have direct access to an Azure app plan, so they can enable/disable any new azure function triggers they build or your slowing everyones development processes down because now every time a developer wants to create a new trigger, they now need to run it past the infrastructure team to add the configuration in for every environment, plus any changes in the future.

The original way this worked with the in-process model allowed developers to create a new azure function triggers and define if it should be running in differernt environments, simply by adjusting the configuration files in the solution. The developers did not need direct access to an azure app plan and thus the CICD could handle the deployment behind the scenes. This works well because both the scheduled timestamps for any azure function trigger jobs and its environmental configuration were both handled in the same place.

Unfortantely, this is the exact reason we currently cannot migrate to the isolated worker model and are stuck on a lower .NET version.

[Schmaga]

I am curious about your current in-proc setup, though. What hosting model are you running on?

We are using the isolated mode hosting model. Or do you mean something else by hosting model?

[fabiocav]

Thanks, @wc-whiteheadd

@Schmaga my question was about the application you're migrating from. You've described what you're doing with your in-proc app, and the challenge getting the same to work with isolated. For the in-proc app, what hosting model are/were you using (consumption, elastic premium, dedicated, etc.)?

[Schmaga]

@Schmaga my question was about the application you're migrating from. You've described what you're doing with your in-proc app, and the challenge getting the same to work with isolated. For the in-proc app, what hosting model are/were you using (consumption, elastic premium, dedicated, etc.)?

Ah, now I understand. We are using a dedicated app service plan for hosting. Have been for the in-proc app, and still are for the isolated mode app.

[luismanez]

Same here. Also migrating to .NET 8 Isolated and this is a blocker for us 😞 (and .net 6 is deprecated ending this year)

[wc-whiteheadd]

I hate to be harsh but is there any updates on this as its technically now been an issue since November 2022, also this ticket has been marked with "enhancement" which is wrong considering its literally blocking people from migrating to it, especially when the .NET 6 deprecation date is coming closer.

Do we have any timescales on when its going to be resolved or the work which is ongoing to solve it?

[JimmyJonesJr]

Stuff like this really adds to the confusion between the isolated and in process models. I've struggled enough finding good documentation for the isolated model, and was really taken aback at how simple functionality like this has been skipped over this far.

[GabrielHSFerreira]

That feature is definitely a blocker. A lot of products are complaining of this to be able to migrate.

[AIZ-THerring]

It would also be nice to have a single configuration value to disable ALL functions in a function app.

I don't want my staging slot, which is running an n-1 version of the function, to be processing my queues.

[kbehboodi]

This is a blocker for us. We use the DisableAttribute to disable our functions on our staging slot. Without this in .NET 8, we cannot upgrade.

[wc-whiteheadd]

just to update, I have now raised this with our MS Azure account manager because there is still no information about the progress on this and the business window is getting smaller to migrate our solutions away from the deprecated version of .NET, leaving us little planning time. The support end date for .NET 6 is "Nov 12, 2024", which is the version most people are limited to until this issue is resolved.

[Ruud-cb]

Ahh... always nice to have a smooth sailing .NET upgrade until you find that ONE single thing that blocks you.

THANKS MICROSOFT. Now I have to split up all the functions for each environment.

@fabiocav please update tags to make this more urgent. Would love to see this as a bug instead of an enhancement, as per every other comment in this Issue.

[gallivantor]

@fabiocav

Our scenario, which I think is quite common, is that our staging deployment slot has an setting TIMERS_DISABLED on it (to stop the timer triggers running in this slot), and then in our function app, all of our 20+ timer triggers are decorated with: [Disable("TIMERS_DISABLED")]

Your proposed approach of requiring a specific environment variable for each function, would introduce a lot of overhead here given that generally, the intent is to disable all the timers in a given deployment slot.

In fact I would think you could deal with 90% of the requests for this by just supporting a standard well-known environment variable called something like FUNCTIONS_DISABLE_TIMERS which, when set, would prevent all timer triggers from firing

[AIZ-THerring]

In fact I would think you could deal with 90% of the requests for this by just supporting a standard well-known environment variable called something like FUNCTIONS_DISABLE_TIMERS which, when set, would prevent all timer triggers from firing

I agree with this sentiment, except I would expand it to include queue triggers, storage triggers, or any other type of trigger that polls or monitors instead of just being a type of webhook.

[elliot-j]

I mocked up a possible solution using middleware that works for timer triggers. In local testing the runtime still tries to invoke the function but the body of the function itself is not reached, so this may still count towards function execution counts for billing purposes

This can probably be extended towards other trigger types, but they may have other requirements like setting specific response values

Program.cs

.ConfigureFunctionsWorkerDefaults((IFunctionsWorkerApplicationBuilder workerApp) => {
    workerApp.UseWhen<DisableFunctionMiddleware>((context) => {
        return DisableFunctionMiddleware.CheckFunctionEnforcement(context);
    });

})

Sample Trigger Function

[Function("TimeFunction")]
public void Run([TimerTrigger("*/30 * * * * *")] TimerInfo myTimer)
{
    _logger.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");

}

Middleware

internal class DisableFunctionMiddleware: IFunctionsWorkerMiddleware
{
    private IGlobalLoggingService _logger;
    public async Task Invoke(FunctionContext context, FunctionExecutionDelegate next)
    {
        ResolveDependencies(context);

        if (!IsTriggerDisabled())
        {
            await next(context);
        }
        else {
            //not calling next(context) will prevent the function from being executed
            _logger.LogInformation($"Function {context.FunctionDefinition.EntryPoint} is disabled");
        }

    }
    private void ResolveDependencies(FunctionContext context)
    {
        _logger = (IGlobalLoggingService)context.InstanceServices.GetService(typeof(IGlobalLoggingService))!;

    }
   //exposed so we only apply the middleware to TimerTrigger functions
    public static bool CheckFunctionEnforcement(FunctionContext context)
    {
        // We want to use this middleware only for http trigger invocations.
        bool isTimerTrigger = context.FunctionDefinition.InputBindings.Values
                        .First(a => a.Type.EndsWith("Trigger")).Type == "timerTrigger";
        return isTimerTrigger;

    }
    private bool IsTriggerDisabled() { 
        return true; //add what ever conditions you want (reflection checks, config settings etc)
    }
}

[SeanFeldman]

@elliot-j This is a good workaround for the timer trigger.

This can probably be extended towards other trigger type

Unfortunately, it won't work for triggers such as service bus trigger, where invocations should not be happening at all. Trying to resolve those from the middleware will cause messages to go into the dead-letter queue, which will cause other problems.

[tbasallo]

@fabiocav @pragnagopa @mumurug-MSFT

The .net6 deadline is fast upon us. This is a huge blocker for us since this adds an additional complication and a solution finding exercise and thus more work than what isolated-mode is already requiring.

This is still considered an enhancement, when it should be a bug or regression since it's something that was working and is no working for a required migration.

There has been plenty of questions and issues related to this. And I am sure there are quite a few orgs that don't even realize this will be an issue because they're procrastinating the migration and are (naively) assuming this will be available.

Can we get an update on this removed feature? What's the plan for supporting it, if any? And what that timeline looks like.

Some related issues:

9229, #9751, #2412, #9693

[waynebrantley]

Perhaps an easy solution would be to let program.cs control the registration/setup of all triggers. Instead of automatically setting up every trigger, let us have some control over it - so in the case of disabling all triggers in a staging slot - would be able to just not register them.

This particular issue has been a blocker of .net isolated since it was announced and the appears to be no workaround at this time short of putting a disabled config setting for every function into the staging slot. This however does not help with development scenarios, etc.

[Zoe1808]

Sharing our workaround...

1. We defined our own attribute: [DisableFunction]

   [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
   public class DisableFunctionAttribute : Attribute, IFilterMetadata
   {
   public DisableFunctionAttribute(string settingName)
   {
       this.SettingName = settingName;
   }
   
   internal string SettingName { get; }
   }

2. As currently the default function indexing is worker indexing, so we added a custom Function metadata provider and ensure this is registered as the last IFunctionMetadataProvider, so we could get the function metadata from the default provider and filter with our attribute. In this case, functions with [DisableFunction] attribute will not be indexed.:

   public class CustomFunctionMetadataProvider : IFunctionMetadataProvider
   {
   private readonly IServiceProvider sp;
   
   public CustomFunctionMetadataProvider(IServiceProvider sp)
   {
       this.sp = sp;
   }
   
   public Task<ImmutableArray<IFunctionMetadata>> GetFunctionMetadataAsync(string directory)
   {
       var service = this.sp.GetServices<IFunctionMetadataProvider>().ToList();
       var functionMetadataProvider = service.Last(x => x.GetType() != typeof(CustomFunctionMetadataProvider));
   
       var metadataList = new List<IFunctionMetadata>();
       Task<ImmutableArray<IFunctionMetadata>> list = functionMetadataProvider.GetFunctionMetadataAsync(directory);
   
       HashSet<string> disabledFunctions = new HashSet<string>();
       var scriptFiles = list.Result.Select(fn => fn.ScriptFile).ToHashSet();
       foreach (var item in scriptFiles)
       {
           Type[] types = Assembly.LoadFrom(item).GetTypes();
           var methods = types.SelectMany(t => t.GetMethods()).Where(m => m.GetCustomAttributes(typeof(DisableFunctionAttribute)).Any() && m.GetCustomAttributes(typeof(FunctionAttribute)).Any());
           foreach (var method in methods)
           {
               string disableSetting = method.GetCustomAttribute<DisableFunctionAttribute>().SettingName;
               if (IsSettingEnabled(disableSetting))
               {
                   disabledFunctions.Add(method.GetCustomAttribute<FunctionAttribute>().Name);
               }
           }
       }
   
       foreach (var item in list.Result)
       {
           if (!disabledFunctions.Contains(item.Name))
           {
               metadataList.Add(item);
           }
       }
   
       return Task.FromResult(metadataList.ToImmutableArray());
   }
   
   private static bool IsSettingEnabled(string settingName)
   {
       // check the target setting and return false (disabled) if the value exists and is "falsey"
       string value = Environment.GetEnvironmentVariable(settingName);
       if (!string.IsNullOrEmpty(value) &&
           (string.Compare(value, "1", StringComparison.OrdinalIgnoreCase) == 0 ||
            string.Compare(value, "true", StringComparison.OrdinalIgnoreCase) == 0))
       {
           return true;
       }
   
       return false;
   }
   }

[tbasallo]

@fabiocav @pragnagopa @mumurug-MSFT @SeanFeldman

Hello team - not trying to be a pain, but the in-process model deadline is getting closer, and this issue has not received enough attention. This issue is detrimental to developers that have functions that kick off with timers or events - what's the proposed solution?

How does Microsoft and the Azure Functions team expect developers to work on a project locally with these triggers firing off? While timers can be handled using a few methods, others like service bus, queues, blobs, etc. will end up in an undesirable state - and probably unbeknownst to them.