Allow External Startup to update the config in all scenarios

[paulhickman-a365]

Issue #6542 resolved a problem where if a functions app modified the configuration object (e.g. to retrieve configuration from KeyVault) compiling with Version 1.1.0 Microsoft.Azure.Functions.Extensions caused the error.

Microsoft.Azure.WebJobs.Script: The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class. 

Compiling with version 1.0.0 of Microsoft.Azure.Functions.Extensions works succesfullly.

The fix applied in #6542 only fixed this regression if the premium app service plan was being used AND runtime scale monitoring was enabled. It does not fix the problem:

• If some other plan is used (e.g. consumption)
• If runtime scale monitoring is not enabled

Thus myself and many other users who have commented on that closed issue are still unable to update to the latest version of the extensions assembly.

[v-anvari]

Hi @paulhickman-a365 , This scenario was explicitly down for certain behaviors, for instance if we enable custom configuration for consumption plan, then function App wouldn't work as expected. As the scale controller wouldn't have access to that information. The scaling would require access to that information as per the current infrastructure. So this scenario wouldn't be possible at this point of time.

Thank you for your feedback, closing this issue.

[mateus-de-castro]

@v-anvari , Is it possible that given that a config is being modified during DI, that you add it to the local.settings.json or the right configuration file? There are scenarios where you don't need to configure it for every instance, just once.

[v-anvari]

Hi @mateus-de-castro , You query is not clear, could you please elaborate the query

[paulhickman-a365]

Hi @paulhickman-a365 , This scenario was explicitly down for certain behaviors, for instance if we enable custom configuration for consumption plan, then function App wouldn't work as expected. As the scale controller wouldn't have access to that information. The scaling would require access to that information as per the current infrastructure. So this scenario wouldn't be possible at this point of time.

Thank you for your feedback, closing this issue.

What about the scenario where you are using premium plan and not using runtime scale monitors?

Not fixing this means our Azure functions costs are going to increase significantly as we will need to leave an instance up 24/7

[mateus-de-castro]

Sorry @v-anvari, I'm going to describe exactly what's my scenario. I'm want to use the CosmosDB trigger but I work with several environments (besides Production and Development) and each one has a different name for the database. Our infrastructure is based on appsettings files - it is loaded during DI based on the environment and among other things it contains the database name - but I can't use it because of this error.

I think that the DI code runs just once, after deployment. Is it possible to generate the local.json file based on the builder configuration? And if it's also impossible because of the current architecture, are there any plans to change it in the next major?

[mateus-de-castro]

@paulhickman-a365, is your problem related to the connection string? If so, maybe this could help you: https://briandunnington.github.io/azure_functions_dynamic_connection_string

[paulhickman-a365]

@paulhickman-a365, is your problem related to the connection string? If so, maybe this could help you: https://briandunnington.github.io/azure_functions_dynamic_connection_string

Thanks for the tip. It is due to a connection string, but it is a service bus connection string rather than a cosmos one. I can't see the equivalent code to use the Options as a backup source of configuration in the binding class for that trigger type - it just passes the parameter to the INameResolver provided by the caller.

[v-anvari]

Hi @paulhickman-a365 , To the question of why we can't implement these changes in consumption plan is here. The nature of consumption is that we scale your applications output 0 so you don't have to pay. When we scale your application down to zero, you application is not available to provide configuration information to the infrastructure.

In Premium plan, the requirement is that the instance is running and at least one instance is warm. So the scale controller always has the access to the config information

In consumption plan, The scale controller does not have access to every possible configuration source and application. Think about the fact that an application can at runtime once it starts up, it could potentially invoke an API. From a different configuration provider that is completely outside of Azure, in order to get all of the connection strings in order to get all of the configuration properties that would apply to the trigger without running that logic, the scale controller, the infrastructure has no way of knowing how to connect to the event source in order to activate the app.

This is a limitation the consumption plan has, and in no plan for a fix.

Let us know if you have any further queries regarding this scenario

[mateus-de-castro]

Hi @v-anvari, around that. In my case I just want to use a static config, that won't change or at least I understand that the instances configuration won't change automatically, if I update a secret on my key vault for example (which is actually the case for the cosmos connection string). I think that right now you want to explicit that behavior so you throw an Error.

The problem is: the way it's implemented even if I'm ok with the configuration being static, I can't use it because of this error. Is it possible to change this error to a warning? Maybe also provide steps on how to trigger an update the current configuration.

I think that you 'calculate' the configuration and save it when the AF is deployed, right?

[v-anvari]

Hi @mateus-de-castro, Are you using the consumption plan as well

[mateus-de-castro]

Yes, I'm using the consumption plan.

[paulhickman-a365]

In Premium plan, the requirement is that the instance is running and at least one instance is warm. So the scale controller always has the access to the config information

@v-anvari I am using the premium plan, but do not have "Use runtime scaling" ticked. I don't think your reasoning for it not working in the consumption plan applies here. Why is this error enforced when using a premium plan without runtime scaling?

[v-anvari]

Hi @paulhickman-a365 , Thank you for your feedback, your query is unclear to us. As you have mentioned in your comments earlier is all about consumption plan. Can you explain the problem with repro steps

What are you trying to achieve where is the error Plan used Expected behavior Actual behavior

If there are issues with premium plan, then we need to investigate the error accordingly.

[v-anvari]

Hi @mateus-de-castro , Your issue seems to be a different issue. If your concern is regarding "Allow External Startup to update the config when Runtime Scale Monitoring is enabled ", then this feature is fixed only for premium plan and cannot be fixed for consumption plan. This is current limitation. If you are seeing any errors in your app which needs to be fixed, please open a new case with all the information in the issue form so as to investigate the problem

[paulhickman-a365]

My goal is to run my function in a premium plan so code inside the function can make calls to databases over a vnet (which consumption doesn't allow), but without the cost overhead of keeping pre-warmed instances which happens when turning on runtime scale monitoring. I also want to store configuration settings in keyvault in such a way that the keyvault secret is used both when running locally on the development VM and run in Azure. This configuration works perfectly with version 1.0.0 of the SDK but fails with 1.1.0 due to the extra validation of the configuration.

Here's how to recreate the issue:

1. In Azure, create a service bus account, basic plan

2. Add a queue called myqueue with default settings and add a shared access policy mysas with "Manage" rights. Copy the primary connection string to your clipboard, paste into notepad and remove the EntityPath part from the end.

3. Create a key vault

4. Add a secret to the vault with the name MyServiceBusTrigger. Set the value to the primary connection string of the shared access policy without the entity path from step 2 above.

5. Create a new function app in Visual Studio 2019.11.0 Preview 3

6. Choose a service bus trigger for the function template called MyServiceBusTrigger with queue name myqueue so that it generates the function:

   
   using System;
   using Microsoft.Azure.WebJobs;
   using Microsoft.Azure.WebJobs.Host;
   using Microsoft.Extensions.Logging;

namespace BrokenFunctionApp { public static class Function1 { [FunctionName("Function1")] public static void Run([ServiceBusTrigger("myqueue", Connection = "MyServiceBusTrigger")]string myQueueItem, ILogger log) { log.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem}"); } } }

7. Add nuget references to the project to

Microsoft.Azure.Functions.Extensions 1.1.0
Microsoft.Extensions.Configuration.AzureKeyVault 3.1.18

8. Update the template's version of service bus to

Microsoft.Azure.Webjobs.Extensions.ServiceBus 4.3.0

9. Add the following ConfigurationInstaller.cs class to the project

using Microsoft.Azure.Functions.Extensions.DependencyInjection; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.Hosting; using System;

namespace BrokenFunctionApp { public static class ConfigurationInstaller { public static IFunctionsHostBuilder AddKeyVault(this IFunctionsHostBuilder builder) { var provider = builder.Services.BuildServiceProvider(); var existingConfig = provider.GetService();

        var hostingEnvironment = provider.GetService<IHostingEnvironment>();
        IConfigurationBuilder configurationBuilder = new ConfigurationBuilder().SetBasePath(hostingEnvironment.ContentRootPath);

        if (existingConfig != null)
        {
            configurationBuilder.AddConfiguration(existingConfig);
        }
        else
        {
            configurationBuilder
                .AddEnvironmentVariables()
                .AddJsonFile("local.settings.json", true, true);
        }

        var currentConfiguration = configurationBuilder.Build();
        var vaultUrl = currentConfiguration["KeyVault"];

        if (string.IsNullOrWhiteSpace(vaultUrl))
        {
            throw new InvalidOperationException("You must provide an Application Setting 'KeyVault' that contains the URL of the keyvault containing the secrets required by this application.");
        }
        //add the key vault to the configuration builder
        configurationBuilder.AddAzureKeyVault(vaultUrl);
        //build the config again so it has the key vault provider
        var config = configurationBuilder.Build();
        //replace the existing config with the new one
        builder.Services.Replace(ServiceDescriptor.Singleton(typeof(IConfiguration), config));

        return builder;
    }
}

}

10. Add a Startup.cs to the project:

using Microsoft.Azure.Functions.Extensions.DependencyInjection;

[assembly: FunctionsStartup(typeof(BrokenFunctionApp.StartUp))] namespace BrokenFunctionApp { public class StartUp : FunctionsStartup { public override void Configure(IFunctionsHostBuilder builder) { builder.AddKeyVault(); }

}

}

11. In local.settings.json add a setting:

"Keyvault": "<URL of the keyvault created in step 3>"

12. In Visual Studio Tools->Options->Azure Service Authentication, select the account that created the function app as the authentication account for Azure

13. Run the app locally in the functions. host. Note that you get warning:

"Warning: Cannot find value named 'MyServiceBusTrigger' in local.settings.json that matches 'connection' property set on 'serviceBusTrigger' in 'C:\Users\Administrator\source\repos\BrokenFunctionApp\BrokenFunctionApp\bin\Debug\netcoreapp3.1\Function1\function.json'. You can run 'func azure functionapp fetch-app-settings ' or specify a connection string in local.settings.json."

Then the warning:

The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class. Function 'Function1' uses the modified key(s): MyServiceBusTrigger

However it just a warning and it does connect to service bus

14. Publish the function app to azure from visual studio, as a Windows function app, premium plan, with a new hosting plan of size EP1. Do not configure keyvault or app insights detected dependencies

15. Goto the keyvault created in step 3 and add an Access Policy with Secret permissions of "List" and "Get". Assign it to the principal for the function app's system identity but do not set the authorized application. Save the policy

16. Goto the function app in Azure portal and in the Configuration -> AppSettings, add setting KeyVault with the URL of the keyvault created in step 3. Confirm that on "Function Runtime settings" tab, the Runtime Scale Monitoring has its default setting of "Off" and save the changes

17. Goto the overview page. If you have a keyvault permissions error, wait minute the restart the app.

18. Click refresh and you see the error at the top of the overview page
````"Functions runtime error
Microsoft.Azure.WebJobs.Script: The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class. Function 'Function1' uses the modified key(s): MyServiceBusTrigger ."

This was only a warning when running locally but breaks the app in Azure.

It can be resolved by turning on "Runtime Scaling Monitoring" in "Function runtime settings" but that requires a pre-warmed VM which increases costs

19. Go back to manage nuget packages in visual studio and roll Microsoft.Azure.Functions.Extensions back to 1.0.0 and Microsoft.NET.Sdk.Functions back to 3.0.9. Republish the application

It now works because these versions of the assemblies don't contain the erroneous validation and shows the application does work in this configuration.

[v-anvari]

Thank you for the details, we will investigate further and update as appropriate.

[v-anvari]

Hi @paulhickman-a365 ,

Here is the reason why you are seeing the error when the Runtime scaling is not enabled. The theory is the same just like consumption plan but the properties are different. The reason the error is enforced is because of the way runtime scaling works, the function runtime scaling has visibility over the configuration and key vault and is doing the work of the scale controller which monitors the scaling. When runtime scaling is enabled, runtime scaling does all the work of a scale controller and it also has a visibility to the secret which is configured for your app. The infrastructure works that way. If you are using premium plan and runtime scaling is not enabled, the runtime scaling would not be able to access the secrets and would not be making the scaling decisions and instead it will be the scale controller and the scale controller will not have the visibility over the secrets so it will not be able to make the decisions. Further explained here - https://github.com/Azure/azure-functions-host/pull/7085

The error is enforced for a reason and disabling this validation would break the infrastructure. Also, the error is enforced on the trigger as explained here - https://github.com/Azure/azure-functions-host/issues/6542 . You can use it on a binding without any error, then its not the trigger who is fetching the key secrets, its the binding used.

[paulhickman-a365]

In my real application, I don't want any scaling to occur at all - my function is a singleton and is marked with [SingletonFunction]. The fact the scale controller can't access the queue isn't a problem in my scenario.

[notaroobob]

I think the not so fun issue is the emulator lets us do these modifications so we can use cool toys like Azure App Configuration service. Then we deploy in the cloud and are met with obstacles that we didn't see coming. With this issue Open, is there going to be a resolution where consumption tiers can take advantage of the Azure Configuration offerings or do we need to accept that some configs will still need to be promoted into the settings?

[Ruud-cb]

I also responded to the referenced issue, yet now facing it again when trying to upgrade to v4. Fun thing though, one function (acceptance) was working perfectly but production not. I used https://resources.azure.com/ to check the config differences.

subscriptions
  -> My Subscription
    -> resourceGroups
      -> My PoC Apps
        -> providers
          -> Microsoft.Web
            -> sites
              -> MyTestApp

I noticed that minimumElasticInstanceCount was set to 1 in production and to 0 in acceptance. After a lot of googling I couldn't find a way to change this in the Azure Portal, so I went on and modified it their.

I have another out-of-memory exception to deal with, error still occurs currently but I hope it's fixed after that, at least it seems to occur less frequent...

Edit: Eventually above didn't solve my problem. I gave up and put the complaining key in azure portal app settings and out of my appsettings.json. That seems to have solved the problem.