WebJob: Setting CosmosDBTrigger connection string(s) at runtime

[tjrobinson]

I would like to be able to configure the CosmosDBTrigger connection string at runtime so that I can build the connection string dynamically, for example so I can use secrets stored in Key Vault.

Repro steps

I have the following method defined, using the CosmosDBTrigger:

public Task ProcessChangeFeed(
            [CosmosDBTrigger(
                databaseName: "mydatabase",
                collectionName: "mycollection",
                ConnectionStringSetting = "CosmosDB:ConnectionString",
                LeaseCollectionName = "leases",
                LeaseDatabaseName = "changefeed",
                LeaseConnectionStringSetting = "CosmosDB:ConnectionString",
                CreateLeaseCollectionIfNotExists = true)]
            IReadOnlyList<Document> documents,
            Microsoft.Extensions.Logging.ILogger logger)
        {

I've tried the following approaches to set the value of the setting CosmosDB:ConnectionString.

Hard-code the value in my appsettings.json file

This is the only approach that works but requires the full connection string, including the AccountKey to be in the config file, or in the Azure Portal (to avoid checking in secrets) - only really an option for local development.

[
  "CosmosDB":
  {
    "ConnectionString" : "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="
  }
}

Using IPostConfigureOptions

This runs, but since the trigger doesn't appear to use IOptions<CosmosDBOptions> it doesn't help. I'm noting it here for completeness.

configurationBuilder.AddAzureKeyVault("https://mykeyvault.vault.azure.net/");

    public class PostConfigureCosmosDBOptions : IPostConfigureOptions<CosmosDBOptions>
    {
        private readonly IConfiguration configuration;

        public PostConfigureCosmosDBOptions(IConfiguration configuration)
        {
            this.configuration = configuration;
        }

        public void PostConfigure(string name, CosmosDBOptions cosmosDbOptions)
        {
            string cosmosDbAccountKey = this.configuration["CosmosDB:AccountKey"];

            string serviceEndpoint = this.configuration["CosmosDB:ServiceEndpoint"];

            string connectionString = $"AccountEndpoint={serviceEndpoint}‌​;AccountKey={cosmosDbAccountKey}";

            cosmosDbOptions.ConnectionString = connectionString;
        }
    }

Adding the setting that the trigger is looking for using AddInMemoryCollection

Again, this code runs and stepping through I can see it adding the correct value but the trigger doesn't seem to use it.

string cosmosDbAccountKey = configurationRoot[KeyVaultSecretKeys.CosmosDbAuthKey];

string serviceEndpoint = configurationRoot[ConfigurationKeys.Data.CosmosDb.ServiceEndpoint];

string connectionString = $"AccountEndpoint={serviceEndpoint}‌​;AccountKey={cosmosDbAccountKey}";

configurationBuilder.AddInMemoryCollection(
    new Dictionary<string, string>
    {
        ["CosmosDB:ConnectionString"] = connectionString
    });

Error information

The error I get if the connection string isn't configured is Request url is invalid which makes sense since it's presumably null/empty.

System.InvalidOperationException: Cannot create Collection Information for Entity in database hrcm with lease leases in database changefeed : Request url is invalid.
ActivityId: 5d22add4-973b-4685-bee0-38b9c08f3e90, Microsoft.Azure.Documents.Common/2.0.0.0, Windows/10.0.17763 documentdb-netcore-sdk/2.1.3 ---> Microsoft.Azure.Documents.DocumentClientException: Request url is invalid.
ActivityId: 5d22add4-973b-4685-bee0-38b9c08f3e90, Microsoft.Azure.Documents.Common/2.0.0.0, Windows/10.0.17763 documentdb-netcore-sdk/2.1.3
   at Microsoft.Azure.Documents.Client.ClientExtensions.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings)
   at Microsoft.Azure.Documents.Client.GatewayServiceConfigurationReader.GetDatabaseAccountAsync(Uri serviceEndpoint)
   at Microsoft.Azure.Documents.Routing.GlobalEndpointManager.GetDatabaseAccountFromAnyLocationsAsync(Uri defaultEndpoint, IList`1 locations, Func`2 getDatabaseAccountFn)
   at Microsoft.Azure.Documents.Client.GatewayServiceConfigurationReader.InitializeReaderAsync()
   at Microsoft.Azure.Documents.Client.DocumentClient.InitializeGatewayConfigurationReader()
   at Microsoft.Azure.Documents.Client.DocumentClient.GetInitializationTask()
   at Microsoft.Azure.Documents.Client.DocumentClient.EnsureValidClientAsync()
   at Microsoft.Azure.Documents.Client.DocumentClient.ReadDatabasePrivateAsync(String databaseLink, RequestOptions options, IDocumentClientRetryPolicy retryPolicyInstance)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass1_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync(Func`1 callbackMethod, Func`3 callShouldRetry, Func`1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync(Func`1 callbackMethod, Func`3 callShouldRetry, Func`1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteAsync(Func`1 callbackMethod, IRetryPolicy retryPolicy, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.Client.DocumentClient.CreateDatabaseIfNotExistsPrivateAsync(Database database, RequestOptions options)
   at Microsoft.Azure.WebJobs.Extensions.CosmosDB.CosmosDBService.CreateDatabaseIfNotExistsAsync(Database database) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.CosmosDB\Services\CosmosDBService.cs:line 48
   at Microsoft.Azure.WebJobs.Extensions.CosmosDB.CosmosDBUtility.CreateDatabaseAndCollectionIfNotExistAsync(ICosmosDBService service, String databaseName, String collectionName, String partitionKey, Int32 throughput) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.CosmosDB\CosmosDBUtility.cs:line 41
   at Microsoft.Azure.WebJobs.Extensions.CosmosDB.CosmosDBTriggerAttributeBindingProvider.TryCreateAsync(TriggerBindingProviderContext context) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.CosmosDB\Trigger\CosmosDBTriggerAttributeBindingProvider.cs:line 141
   --- End of inner exception stack trace ---
   at Microsoft.Azure.WebJobs.Extensions.CosmosDB.CosmosDBTriggerAttributeBindingProvider.TryCreateAsync(TriggerBindingProviderContext context) in C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions.CosmosDB\Trigger\CosmosDBTriggerAttributeBindingProvider.cs:line 146
   at Microsoft.Azure.WebJobs.Host.Triggers.CompositeTriggerBindingProvider.TryCreateAsync(TriggerBindingProviderContext context) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Triggers\CompositeTriggerBindingProvider.cs:line 22
   at Microsoft.Azure.WebJobs.Host.Indexers.FunctionIndexer.IndexMethodAsyncCore(MethodInfo method, IFunctionIndexCollector index, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Indexers\FunctionIndexer.cs:line 190
   at Microsoft.Azure.WebJobs.Host.Indexers.FunctionIndexer.IndexMethodAsync(MethodInfo method, IFunctionIndexCollector index, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Indexers\FunctionIndexer.cs:line 167

Expected behaviour

We use the ServiceBusTriggerAttribute and configure the connection string in an implementation of IPostConfigureOptions<ServiceBusOptions> and this works fine. The CosmosDBTrigger behaves differently and doesn't appear to support this.

Related information

• TargetFramework: netcoreapp2.1
• Microsoft.Azure.WebJobs 3.0.1
• Microsoft.Azure.WebJobs.Extensions.CosmosDB 3.0.1
• Using generic HostBuilder in Program.Main()
• Cosmos DB Emulator 2.0.0

[tjrobinson]

Another approach, which also doesn't seem to help:

[assembly: WebJobsStartup(typeof(Startup))]
namespace Empactis.CaseManager.AzureFunctions
{
    internal class Startup : IWebJobsStartup
    {
        public void Configure(IWebJobsBuilder builder)
        {
            builder.AddCosmosDB(
                options => options.ConnectionString = "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==");
        }
    }
}

[tjrobinson]

Could this be moved to https://github.com/Azure/azure-webjobs-sdk-extensions/ ?

[tjrobinson]

Response from the Cosmos DB team:

Dynamic resolution of values in the Functions Bindings and Triggers (in all Bindings and Triggers, not only Cosmos DB) is not currently supported.

If you work with Webjobs though, it should be possible to wire up a KeyVault configuration provider to read your secrets from KeyVault and populate that configuration. I’ve not actually tried it, but here’s docs on wiring it up: https://docs.microsoft.com/en-us/aspnet/core/security/key-vault-configuration?view=aspnetcore-2.1.

[ThinkFr33ly]

Has a work around for this been determined? The idea that we're forced to store secrets in config files or management them in the function portal experience is a deal breaker. We store all secrets in key vault. No exceptions. So unless we can configure this at runtime, we're totally blocked.

[tjrobinson]

@ThinkFr33ly This preview feature may help you: https://azure.microsoft.com/en-us/blog/simplifying-security-for-serverless-and-web-apps-with-azure-functions-and-app-service/

See also: https://docs.microsoft.com/en-gb/azure/app-service/app-service-key-vault-references

[ThinkFr33ly]

@ThinkFr33ly This preview feature may help you: https://azure.microsoft.com/en-us/blog/simplifying-security-for-serverless-and-web-apps-with-azure-functions-and-app-service/

See also: https://docs.microsoft.com/en-gb/azure/app-service/app-service-key-vault-references

@tjrobinson this is great. Had no idea you could do that. I actually managed to get this working using the web job startup class and just setting the configuration value manually. I prefer this method because it keeps my configuration system consistent across all aspects of my app - .config or .json files never have secrets, they only have "secret URIs" and then I grab those using a secret resolver / key vault. This leverages the MSI (probably exactly how your example is doing it).

[mjpandian]

The approach mentioned here requires the Secret URI to be used. https://azure.microsoft.com/en-us/blog/simplifying-security-for-serverless-and-web-apps-with-azure-functions-and-app-service/.. In our case we don't have access to the Keyvault in prod to get the URI. Are there any options to set the Cosmos DB Trigger Connectionstringsetting through Webjobs ?

[jrowies]

@ThinkFr33ly This preview feature may help you: https://azure.microsoft.com/en-us/blog/simplifying-security-for-serverless-and-web-apps-with-azure-functions-and-app-service/ See also: https://docs.microsoft.com/en-gb/azure/app-service/app-service-key-vault-references

@tjrobinson this is great. Had no idea you could do that. I actually managed to get this working using the web job startup class and just setting the configuration value manually. I prefer this method because it keeps my configuration system consistent across all aspects of my app - .config or .json files never have secrets, they only have "secret URIs" and then I grab those using a secret resolver / key vault. This leverages the MSI (probably exactly how your example is doing it).

@ThinkFr33ly would you mind sharing some details about how you implemented your solution? Thanks!

[briandunnington]

Here is how I did it using the PostConfigure options - note that you have to make sure the ConnectionStringSetting is set to "" or else the dynamically set value wont be used: https://briandunnington.github.io/azure_functions_dynamic_connection_string