Cannot deploy to a function hosted on Linux Consumption plan when AzureWebJobsStorage config setting is Key Vault reference

rjygraham commented 3 years ago

When a function hosted on Linux Consumption plan has AzureWebJobsStorage configuration setting set to valid Key Vault reference, errors block successful deployment via Az CLI and Functions CLI.

Investigative information

Please provide the following:

Timestamp: 2021-03-26 04:06:00 UTC
Function App version: ~3 runtime, dotnet-isolated (.NET 5)
Function App name: oe-twitch-eus-identity-2021-03-15-func
Function name(s) (as appropriate): N/A
Invocation ID: N/A
Region: East US

Repro steps

Create Function App using Linux Consumption plan
Enable Function System-Assigned Managed Identity
Create Azure Key Vault and grant Function App identity Secret List/Get permissions in Key Vault ACL
Add Key Vault secret containing the Function's Storage Account connection string as a value
Update the Function App AzureWebJobsStorage to be a Key Vault reference
Attempt to deploy a function app via Az CLI or Functions CLI.

Expected behavior

Step 6 will successfully deploy the code to the Function App

Actual behavior

Az CLI fails with the following errors:

az functionapp deployment source config-zip -g OE-TWITCH-EUS -n oe-twitch-eus-identity-2021-03-15-func --src .\bin\publish\publish.zip
The command failed with an unexpected error. Here is the traceback:
You need to provide an account name and either an account_key or sas_token when creating a storage service.
Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-taryuzhh\knack\cli.py", line 233, in invoke
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 660, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 723, in _run_jobs_serially
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 716, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-l7ikczg1\six.py", line 703, in reraise
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 694, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\commands\__init__.py", line 331, in __call__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-sc_lkrar\azure\cli\core\__init__.py", line 808, in default_command_handler
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-ibh5mnup\azure\cli\command_modules\appservice\custom.py", line 385, in enable_zip_deploy_functionapp
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-ibh5mnup\azure\cli\command_modules\appservice\custom.py", line 543, in upload_zip_to_storage
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\blob\blockblobservice.py", line 146, in __init__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\blob\baseblobservice.py", line 209, in __init__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\common\_connection.py", line 110, in get_service_parameters
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\common\_connection.py", line 161, in _from_connection_string
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-r6yjb8kq\azure\multiapi\storage\v2018_11_09\common\_connection.py", line 85, in __init__
ValueError: You need to provide an account name and either an account_key or sas_token when creating a storage service.
To open an issue, please run: 'az feedback'

Functions CLI fails with the following errors:

func azure functionapp publish oe-twitch-eus-identity-2021-03-15-func
Microsoft (R) Build Engine version 16.9.0+57a23d249 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
  func-dotnet5 -> C:\Users\ryan\source\repos\func-dotnet5\bin\publish\func-dotnet5.dll
  Determining projects to restore...
  Restored C:\Users\ryan\AppData\Local\Temp\ure1mi1o.wlg\WorkerExtensions.csproj (in 1.14 sec).
  WorkerExtensions -> C:\Users\ryan\AppData\Local\Temp\ure1mi1o.wlg\buildout\Microsoft.Azure.Functions.Worker.Extensions.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:08.79

Getting site publishing info...
Uploading package...
Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid
Retry: 1 of 3
Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid
Retry: 2 of 3
Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid
Retry: 3 of 3
Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid

Known workarounds

Do not use a Key Vault reference for AzureWebJobsStorage setting on a Linux Consumption plan
Use a Windows Consumption plan appears to work with AzureWebJobsStorage as Key Vault reference

Related information

Provide any related information

Programming language used: .NET 5
Links to source: N/A
Bindings used: N/A

v-bbalaiagar commented 3 years ago

Hi @balag0, Could you please look into this issue. Transferring this issue to core tools for further investigation.

balag0 commented 3 years ago

Thanks. yes. this is not supported in core tools currently. will take care of this. assigned this to myself cc @pragnagopa

pragnagopa commented 3 years ago

@balag0 - Assigned this current sprint. Let me know if this needs to be assigned to next sprint instead.

fabiocav commented 3 years ago

Assigning this to sprint 101

balag0 commented 3 years ago

Yes, didn't get a chance to work on this. Sprint 101 is good. Thanks

fabiocav commented 3 years ago

@balag0 moving this back to triaged, please feel free to assign to a sprint when we're pretty confident we'll be able to close this. Thanks!

balag0 commented 3 years ago

I am working on this, but this needs some other pieces implemented first in kudu and elsewhere which I am focusing on first.

ChrisProlls commented 3 years ago

Any news on this ?

cveld commented 2 years ago

Just tested, Func cli still fails. Error message: Error creating a Blob container reference. Please make sure your connection string in "AzureWebJobsStorage" is valid

Interestingly deploying from visual studio 2022 (17.2) works fine.

Related, the Azure devops pipeline task fails. Unbelievable I thought I was doing something wrong here. I was also misled by the message in Portal:

The portal is not able to confirm the status of your Key Vault reference at this time. Please confirm the status directly from the app by checking whether the environment variables have resolved.

Just opened a ticket: https://github.com/microsoft/azure-pipelines-tasks/issues/16749

bw-flagship commented 1 year ago

This is crucial, why is it not touched for years? Also, there is no good error message or notification that tells the developer that this is not supported. I wasted hours here!

Azure / azure-functions-core-tools