Expose configuration from worker processes

[Prabhu-dev-alt]

I am trying to connect my function app to keyvault and get queue name and connection secrets. This was working well with .netcore3.1 app using the ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder) method in FunctionStartup.

After upgrading to .net5 dotnet-isolated, the bindings does not work. I configured azurekeyvault in Program.cs but still it does not pick from keyvault. Seems like functionstartup is trying to bind before Main() is invoked.

QueueFunction

 public static void Run([QueueTrigger("%QueueName%", Connection = "QueueConnection")] string message, string id)

Startup.cs (.netcore3.1)- working

 public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
    {
        var azureKeyVaultURL = Environment.GetEnvironmentVariable("AzureKeyVaultURL");
        var azureKeyVaultADAppID = Environment.GetEnvironmentVariable("AzureKeyVaultMIAppID");

        builder.ConfigurationBuilder
                    .SetBasePath(Environment.CurrentDirectory)
                    .AddAzureKeyVault(new Uri(azureKeyVaultURL), new ManagedIdentityCredential(azureKeyVaultADAppID))
                    .AddEnvironmentVariables()
                .Build();
    }

Program.cs (.net5)- Not working

var host = new HostBuilder()
            .ConfigureFunctionsWorkerDefaults()
            .ConfigureAppConfiguration(config =>{
                 var azureKeyVaultURL = Environment.GetEnvironmentVariable("AzureKeyVaultURL");
                var azureKeyVaultADAppID = Environment.GetEnvironmentVariable("AzureKeyVaultMIAppID");

                config
                   .SetBasePath(Environment.CurrentDirectory)
                   .AddAzureKeyVault(new Uri(azureKeyVaultURL), new ManagedIdentityCredential(azureKeyVaultADAppID))
                   .AddEnvironmentVariables()
                .Build();
            })
.Build()

[v-bbalaiagar]

Hi @Prabhu-dev-alt, Transferring this issue for more insights

[twentytwokhz]

@Prabhu-dev-alt Yep, I have the same problem. I'm using AppConfiguration connected to a keyvault though. The host seems to initialize with errored functions. Those functions cannot find their secrets for the bindings...even though I tested and the keys are retrieved after the fact correctly

[Prabhu-dev-alt]

@twentytwokhz Can you provide some example code that you used for connecting the keyvault. Is it in program.cs?

[brettsam]

As you've found, this behavior is different in the isolated worker. Using KeyVault settings in your worker process only apply to that process and are not accessible to the host.

The recommended way to use Key Vault for language workers is to use App Service Key Vault references which will expose settings via App Settings: https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references.

I'm going to leave this open, though and rename it. I wonder if there's a way we can allow Language Workers to expose configuration settings back to the Host on startup, during the initial handshake. This would be a larger work item and I don't even have a guess at an ETA right now.

[lohithgn]

@brettsam is this the same case for even UserSecrets. I am trying to add UserSecrets in ConfigureAppConfiguration. But has no effect.

.ConfigureAppConfiguration((hostContext, builder) => {
                    Console.WriteLine($"hostContext.HostingEnvironment.IsDevelopment():{hostContext.HostingEnvironment.IsDevelopment()}");
                    if (hostContext.HostingEnvironment.IsDevelopment())
                    {
                        builder.AddUserSecrets<Program>();
                    }
                })

Whats the guideline on this front for dotnet-isolated fx?

[PierrickBlons]

Hi all,

Joining the discussion to be sure I'm facing the same issue.

As everyone here I'm trying to load the configuration for my bindings from AzureKeyVault.

I realized that the problem wasn't coming from the AddAzureKeyVault extension for IConfigurationBuilder but rather how the Bindings attributes are resolving their Connection property. Accessing the configuration from the IConfiguration interface injected in the constructor is working well in the Run function but not for the bindings.

What can be done to have the binding attributes picking up the configuration from the IConfiguration ?

Here is the repro code

Program.cs

public class Program 
{ 
    public static void Main()
    {
        var host = new HostBuilder()
            .ConfigureAppConfiguration(config => {               
                config.AddInMemoryCollection(new Dictionary<string, string>()
                {
                   {"InMemoryConnection", "myConnectionStringHere" },
                });
            })
            .ConfigureFunctionsWorkerDefaults()
            .Build();
        host.Run();
    }
}

QueueTriggerCSharp1.cs

public class QueueTriggerCSharp1
{
    private readonly IConfiguration _configuration;
    public QueueTriggerCSharp1(IConfiguration configuration)
    {
        _configuration = configuration;
    }

    [Function("QueueTriggerCSharp1")]
    public void Run([QueueTrigger("myqueue-items")] string myQueueItem,
        FunctionContext context)
    {
        var logger = context.GetLogger("QueueTriggerCSharp1");
        string connectionStringFromConfiguration = _configuration.GetValue<string>("InMemoryConnection");
        logger.LogInformation($"Connection from IConfiguration: {connectionStringFromConfiguration}"); // Can read the configuration properly here
    }
}

Output :

[2021-05-20T11:08:40.777Z] Worker process started and initialized.
[2021-05-20T11:08:45.411Z] Host lock lease acquired by instance ID '00000000000000000000000038F64B19'.
[2021-05-20T11:09:06.557Z] Executing 'Functions.QueueTriggerCSharp1' (Reason='New queue message detected on 'myqueue-items'.', Id=1d5a492c-1895-4ac8-a631-ea64f35eea2a)
[2021-05-20T11:09:06.559Z] Trigger Details: MessageId: 01d4e0ff-7b02-4bf5-bf59-9b8a07c53e23, DequeueCount: 1, InsertionTime: 2021-05-20T11:09:05.000+00:00

[2021-05-20T11:09:06.680Z] Connection from IConfiguration: myConnectionStringHere

[2021-05-20T11:09:06.730Z] Executed 'Functions.QueueTriggerCSharp1' (Succeeded, Id=1d5a492c-1895-4ac8-a631-ea64f35eea2a, Duration=174ms)

But when using the following QueueTriggerAttribute definition like this

public void Run([QueueTrigger("myqueue-items", Connection = "InMemoryConnection")] string myQueueItem,
            FunctionContext context)

Output :

[2021-05-20T11:34:31.711Z] Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.QueueTriggerCSharp1'. Microsoft.Azure.WebJobs.Extensions.Storage: Storage account connection string 'AzureWebJobsInMemoryConnection' does not exist. Make sure that it is a defined App Setting.
[2021-05-20T11:34:31.714Z] Error indexing method 'Functions.QueueTriggerCSharp1'
[2021-05-20T11:34:31.715Z] Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.QueueTriggerCSharp1'. Microsoft.Azure.WebJobs.Extensions.Storage: Storage account connection string 'AzureWebJobsInMemoryConnection' does not exist. Make sure that it is a defined App Setting.
[2021-05-20T11:34:31.716Z] Function 'Functions.QueueTriggerCSharp1' failed indexing and will be disabled.
[2021-05-20T11:34:31.746Z] The 'QueueTriggerCSharp1' function is in error: Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.QueueTriggerCSharp1'. Microsoft.Azure.WebJobs.Extensions.Storage: Storage account connection string 'AzureWebJobsInMemoryConnection' does not exist. Make sure that it is a defined App Setting.

To be sure it was not only applying to InputTrigger I tested also on an HttpTrigger function with a QueueOutput :

[Function("HttpExample")]
[QueueOutput("myqueue-items", Connection = "InMemoryConnection")]
public HttpResponseData Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
    FunctionContext executionContext)

Output when calling the URIs

[2021-05-20T11:37:31.487Z] System.Private.CoreLib: Exception while executing function: Functions.HttpExample. Microsoft.Azure.WebJobs.Host: Storage account connection string 'AzureWebJobsInMemoryConnection' does not exist. Make sure that it is a defined App Setting. 

After some digging in the source code of the dotnet worker I've noticed that the version 4.0.4 of the Microsoft.Azure.WebJobs.Extensions.Storage package is used.

Recently on the 5.0.0-beta4 some updates has been made on how the configuration is handled for the creation of the QueueClient from the QueueTriggerAttribute.

https://github.com/Azure/azure-sdk-for-net/commit/0677c048170785eb4793cefb57771468af295b5f

Do you think upgrading the functions dotnet worker to use the version 5.0.0 can help to pickup the configuration from the AppConfiguration?

[rohinz]

All customization in Program.cs applied to the IConfigurationBuilder will just affect the isolated worker process. But the function host (which is another process) will not be affected. And this causes issues with Input- and Output binding configurations.

@lohithgn: KeyVaultReferences are not a real alternative as long as they are only supported with system-assigned identities but not with user-assigned identities.

Exposing the configuration from worker to host is one option. But having a better integration with App Configuration Service could also be an option?

[lohithgn]

@rohinz I am not trying with Key Vault.

My scenario was: during development - I don't want my team to keep secrets in local.settings.json. So we use User Secrets configured on the project. When we moved to .NET5 based dotnet-isolated model for functions - we thought the user secrets were not getting loaded. The code we had used was hostContext.HostingEnvironment.IsDevelopment(). IsDevelopment was always false for us. This is because - at the moment the only way you can run .NET 5 based function apps is through command line. func start --dotnet-isolated-debug.

Since the .NET 5 based fx uses generic host to bootup - I am not sure what environment variable needs to be set to let host know that it is in Development mode now. Usually Visual Studio does the magic of injecting environment variable when we Run the project. So we utilized one of the well know tricks to check if we are in Development or production. If we are in development - we inject User Secrets - in prod we don't.

Also I noticed that the order in which we set up the host matters. We ended up with the following sequence and it works for us. We have only Http triggers for now.

var host = new HostBuilder()
                .ConfigureFunctionsWorkerDefaults()
                .ConfigureAppConfiguration((hostContext, builder) => {
                    var environment = Environment.GetEnvironmentVariable("WEBSITE_SLOT_NAME") ?? "Development";
                    if (environment == "Development")
                    {
                        builder.AddUserSecrets<Program>();
                    }
                })
                .ConfigureServices(services => Startup.ConfigureServices(services))
                .Build();

Hope this helps.

[twentytwokhz]

For me this is really frustrating. I am setting up the Azure App Configuration Service in startup and I'm having all sorts of weird behaviors where some configuration values are available and some are not. I assume the ones in startup are read automatically when the code is executed..but this code is executed only after the functions are initiated - thus, functions are lacking their backing configuration support.

var host = new HostBuilder()
                .ConfigureAppConfiguration(c =>
                {
                    // Add Environment Variables since we need to get the App Configuration connection string from settings.
                    c.AddEnvironmentVariables();

                    // Call Build() so we can get values from the IConfiguration.
                    var config = c.Build();
                    var appcsUrl = config.GetValue<string>("AppConfigUrl");

                    // Add Azure App Configuration with the connectionstring.
                    c.AddAzureAppConfiguration((options) =>
                    {
                        var defaultAzureCredential = GetDefaultAzureCredential();

                        options.Connect(new Uri(appcsUrl), defaultAzureCredential)
                            // also setup key vault for key vault references
                            .ConfigureKeyVault(kvOptions =>
                            {
                                kvOptions.SetCredential(defaultAzureCredential);
                            });
                        options.Select(KeyFilter.Any, LabelFilter.Null);
                    });
                })
                .ConfigureFunctionsWorkerDefaults()
                .ConfigureServices((context, services) =>
                {
                    services.AddBreathBaseDbContext(context.Configuration["DB-ConnectionString"]);

                    services.AddTransient<ITableService, TableService>();

                    services.AddSingleton(RegistryManager.CreateFromConnectionString(context.Configuration["IoTHub-ConnectionString"]));
                    services.AddSingleton<IIoTHubService, IoTHubService>();

                    services.AddSingleton<CosmosClient>(new CosmosClient(context.Configuration["CosmosDB-ConnectionString"]));
                    services.AddTransient<ICosmosDbService, CosmosDbService>();
                })
                .Build();

This actually forces me to setup keyvault references on the actual function app, instead of relying solely on the Azure App Configuration service

[brettsam]

@twentytwokhz -- nothing you're doing here is specific to Functions. Function classes are instantiated using the standard .NET dependency injection -- so all configuration and options should be initialized at that time. I suspect that your usage of the configuration straight from the context in ConfigureServices may be the issue.

If you can share an example of where the configuration is not yet set up when a function is instantiated, I may be able to give a more concrete recommendation.

edit -- I suspect that if you used factories to create your services, it may behave the way you want. For example:

services.AddSingleton<CosmosClient>(s =>
{
    var config = s.GetService<IConfiguration>();
    return new CosmosClient(config["CosmosDB-ConnectionString"]);
}); 

[twentytwokhz]

@brettsam What I currently see is that the connection ServiceBus-ConnectionString in my following function is not being populated from the App Service Configuration.

Could it be related to the DI of the services I configured in the ConfigureServices?

public class Test_ServiceBusTrigger
    {
        private readonly ICosmosDbService cosmos;
        private readonly IIoTHubService ioTHubService;
        private readonly ILogger<SyncDevices_ServiceBusTrigger> log;

        public Test_ServiceBusTrigger(ICosmosDbService cosmos,
            IIoTHubService ioTHubService, ILogger<Test_ServiceBusTrigger> log)
        {
            this.cosmos = cosmos;
            this.ioTHubService = ioTHubService;
            this.log = log;
        }

        [Function("Test_ServiceBusTrigger")]
        public async Task Run([ServiceBusTrigger("test-queue", Connection = "ServiceBus-ConnectionString")] string myQueueItem)
        {
            log.LogInformation($"SyncDevices_ServiceBusTrigger queue trigger function processed message: {myQueueItem}");

            var devices = await ioTHubService.GetAllDevicesAsync();
            //redacted
        }
    }

Screenshot from local debug session

[ianmccaul]

Is there a timeframe to get a fix for this? We rely heavily on App Configuration to set all the config values, this worked in prior function versions and is going to make it difficult to migrate to .net 5 and onward.

[twentytwokhz]

Any updates on this? It's been more than 4 months now

[lpunderscore]

This is absolutely horrible. Completely breaking and undocumented. We now have to expose connection strings directly in local files as we can't use keyvault or app configuration for bindings etc.

You need to document this limitation, and warn people against using isolated functions in production until this is resolved.

This should be a TOP priority and certainly not flagged as enhancement... who does triage on these.... jesus.

[CezaryKlus]

I'm another one struggling with this issue. There needs to be a bridge to the host process enabled. Why not reuse the concept of in-process FunctionsStartup?

[ArildEik]

I'm not sure if this is solved yet, but I had the same problem with my isolated Azure function earlier today. Everything worked locally at my machine, but as soon as I put it in Azure, configuration was not available during startup (program.cs). My solution that seems to work, was to load the configuration separately those places I needed to check config-settings.

The code below supports reading config settings from local.settings.json, User Secrets json file and Azure Key Vault when running from local machine, and from Azure Function Settings and Azure Key Vault when running in Azure.

The main solution to my problem earlier today, was the LoadConfigurations() method. The two methods from CrmServiceCommon are both returning boolean.

Here is my Program.cs. I hope you can use some of it...

var host = new HostBuilder()
 .UseServiceProviderFactory(new AutofacServiceProviderFactory())
 .ConfigureFunctionsWorkerDefaults();

host.ConfigureContainer<ContainerBuilder>(containerBuilder =>
{
  containerBuilder.RegisterModule(new DefaultCoreModule());
  containerBuilder.RegisterModule(new DefaultInfrastructureModule(CrmServiceCommon.EnvironmentIsDevelopment()));
});

// Configure the "global" configuration service returned by DI
host.ConfigureAppConfiguration((hostContext, builder) =>
{
  // Add the use of User Secret JSON file 
  if (CrmServiceCommon.EnvironmentIsDevelopment())
    builder.AddUserSecrets(Assembly.GetExecutingAssembly(), true);

  // Check if we need to add Azure Vault support
  var useAzureKeyVaultLocally = CrmServiceCommon.UseAzureKeyVaultLocally(); 

  // Exit if we don't need to add KeyVault support
  if (CrmServiceCommon.EnvironmentIsDevelopment() && !useAzureKeyVaultLocally) return;

  // Add support for Kay Vault
  var keyVaultUri = GetKeyVaultUri();
  builder.AddAzureKeyVault(keyVaultUri, GetKeyVaultCredentials());
});

// Configure DBContext
host.ConfigureServices((hostContext, services) =>
{
  if (CrmServiceCommon.EnvironmentIsDevelopment())
  {
    var config = LoadConfigurations(false, true);
    services.AddDbContext(config["CrmServicePortal:DatabaseConnectionString"]);
  }
  else
  {
    var config = LoadConfigurations(true, false, true);
    services.AddDbContextWithNoTracking(config["CrmServicePortal:DatabaseConnectionString"]);
  }
});

var app = host.Build();
app.Run();

static IConfigurationRoot LoadConfigurations(bool includeDefaultEnvironment, bool includeUserSecrets = false, bool includeAzureKeyVault = false)
{
  // Create new config builder definition
  var configurationDefinition = new ConfigurationBuilder().AddInMemoryCollection();

  // Add support for local.settings.json and default environment variables
  if (includeDefaultEnvironment)
    configurationDefinition.AddEnvironmentVariables();

  // Add support for User Secrets
  if (includeUserSecrets)
    configurationDefinition.AddUserSecrets(Assembly.GetExecutingAssembly(), true);

  // If we don't want Azure Key vault settings, build and return
  if (!includeAzureKeyVault) return configurationDefinition.Build(); 

  // Add support for Key Vault
  var keyVaultUri = GetKeyVaultUri();
  configurationDefinition = configurationDefinition.AddAzureKeyVault(keyVaultUri, GetKeyVaultCredentials());

  return configurationDefinition.Build();
}

static DefaultAzureCredential GetKeyVaultCredentials()
{
  var isDevelopment = CrmServiceCommon.EnvironmentIsDevelopment();

  return new DefaultAzureCredential(new DefaultAzureCredentialOptions
                                    {
                                      ExcludeEnvironmentCredential = true,
                                      ExcludeInteractiveBrowserCredential = true,
                                      ExcludeAzurePowerShellCredential = true,
                                      ExcludeSharedTokenCacheCredential = true,
                                      ExcludeVisualStudioCodeCredential = true,
                                      ExcludeVisualStudioCredential = !isDevelopment,
                                      ExcludeAzureCliCredential = true,
                                      ExcludeManagedIdentityCredential = isDevelopment,
                                      Retry =
                                      {
                                        Delay = TimeSpan.FromSeconds(2),
                                        MaxDelay = TimeSpan.FromSeconds(16),
                                        MaxRetries = 5,
                                        Mode = RetryMode.Exponential
                                      }
                                    });
}

static Uri GetKeyVaultUri()
{
  // Get KayVault URL from local.settings.json/Azure Application Settings
  var keyVaultUrl = Environment.GetEnvironmentVariable("AzureKeyVault:Uri") ?? "";
  if (!CrmServiceCommon.EnvironmentIsDevelopment()) return new Uri(keyVaultUrl);

  // If we are running on local machine, get URL from User Secrets
  var configRoot = LoadConfigurations(false, true);
  keyVaultUrl = configRoot["AzureKeyVault:Uri"];

  return new Uri(keyVaultUrl);
}

[ianmccaul]

Well this will prevent any upgrade to .NET 7 as my understanding is its unlikely to get an in-process version and msft seems to be moving away from inproc and to isolated. Ok, so we cant do that until this is resolved. Having to put connection strings in local dev environments is a risk and management pain when they need to be changed. Especially with IaC involved, all this stuff is provisioned in Azure automatically and wired into App Config. Its been a year since my last comment on this thread, and still no resolution, not even a comment from the team.

[Lillskogen]

Any progress on this?

We are having the same issues as others are mentioning here. We don't want secrets in our local.settings.json so we load these from Azure Key Vault. The host is failing before it is started since connection strings cannot be fetched from the local settings file.

How should the local development scenario be handled in this situation? Seems like a really poor DX if all developer needs to maintain their local settings file. The possibility of adjusting the host process configuration during development would be desirable.

[udlose]

@brettsam Is there any update on this? I'm experiencing the same issue.

[sean09lee]

Any update on this? I am running into this with the new .net 8 isolated worker process v4 function and app configuration. Like everyone else, I can successfully retrieve items from the App Configuration, but all those values are automatically stored in the hostContext object under an App Configuration provider. I have a function that uses % binding expressions - and it cannot seem to read from that specific provider.

Here are some snippets of my setup:

Program.cs

var builder = new HostBuilder();

builder.ConfigureFunctionsWorkerDefaults();

builder.ConfigureAppConfiguration((hostContext, config) =>
{
    var configuration = hostContext.Configuration;
    var connectionString = configuration[AzureAppConfigurationConnectionString];
    var environmentLabel = configuration[AzureAppConfigurationLabel];
    config.AddAzureAppConfiguration(options =>
    {
        options.Connect(connectionString)
            .Select(KeyFilter.Any) //Pulls all without a label
            .Select(KeyFilter.Any, environmentLabel)
            .ConfigureKeyVault(kv => kv.SetCredential(new DefaultAzureCredential()));
    });
});

var host = builder.Build();
await host.RunAsync();

Function.cs

[Function(nameof(BookedNewAsync))]
public async Task BookedNewAsync([TimerTrigger("%ReservationNotifications:BookingCron%")] TimerInfo timer)
{
    await InitiateAsync(ReservationNotificationJobName.BookedNewReservationNotificationBackgroundJob, timer);
}

Resulting error:

This works if I move all my values from App Config into my local.settings.json, but it makes for a horrible developer experience.

[ac931274]

I had an issue a little like this. I wrote about it here: https://stackoverflow.com/questions/78897966/azure-functions-trigger-variables-isolated-functions/78897967

Hope it might help.