Invoking /functions/admin/download from ARM stopped working

[dany74q]

At some, relatively recent, point in time - invoking the /functions/admin/download SCM API for Windows function apps from ARM (https://management.azure.com/subscriptions/<sub>/resourceGroups/<rg>/provider/Microsoft.Web/sites/<site>/functions/admin/download?api-version=2022-09-01) had stopped working - it consistently returns 401 even if granted a Contributor role, or the Microsoft.Web/sites/publish/action​ permission explicitly.

It works as expected with the same authorization token when invoked directly on the SCM site, but I have strict outbound network policies requiring communication with ARM (management.azure.com) alone, so I can't use that.

I was wondering if you know whether anything had changed in that regard, and if so - do you happen to know if there's an appropriate alternative for invoking this API through ARM ?

StackOverflow link: https://stackoverflow.com/questions/75853352/invoking-functions-admin-download-from-arm-stopped-working

[bhagyshricompany]

Hi @dany74q thanks for informing. There have been some recent changes to Azure authentication and authorization mechanisms that may be causing the 401 error when invoking the /functions/admin/download SCM API for Windows function apps from ARM.

One possible solution would be to use the Azure CLI or PowerShell to invoke the SCM API instead of directly calling the API endpoint. This can be achieved by using the az functionapp deployment source config-zip command or the New-AzWebAppZipDeployment PowerShell cmdlet, respectively. These commands allow you to deploy a ZIP package to a function app's SCM site, which in turn triggers a deployment and makes the package available through the /functions/admin/download endpoint.

[dany74q]

Hey @bhagyshricompany, thanks for the prompt response !

I was using said endpoint to retrieve the existing content's of an app, rather than deploying new content, my goal is to be a reader, rather than a publisher, which was possible prior to said change.

Where can I find more info regarding said change - any chance the code is open source and we can link to a commit, or a doc page otherwise ?

Thanks !

[bhagyshricompany]

ok.I will discuss with Team and update you for this Api.

[bhagyshricompany]

@fabiocav pls comment

[FinVamp1]

Hello,

This was a recent security change for Functions and the PR for this is here. PR For Kudu

This is by design and is related to a recent security change for ARM. This will not be reverted and for access to the web site content you should still be able to access this using the Deployment credentials from the publishing profile and use either FTPS or The VFS API.

Let me know if you have other questions.

Thanks,

Finbar

[dany74q]

Hey @FinVamp1 !

Thanks for the response - but I think my issue goes deeper than that, the code change linked above doesn't explain the behavior I'm seeing ]:

The download API doesn't work from ARM for service accounts that are Contributors on that app-service, according to the code in Kudu, this should not be the case, as the rbac contributor header is checked on the bridge http headers: https://github.com/projectkudu/kudu/blob/6db3a01054ee11f9fa888a11281fd2f7d5adb915/Kudu.Services.Web/Tracing/TraceModule.cs#L75-L79

httpRequest.Headers[Constants.RoleBasedContributorHeader] != "1"

I have a Contributor-level principal that fails to invoke the download API, but succeeds invoking the token and masterkey APIs - also, I can see the x-ms-client-rolebased-contributor set to 1 for other requests using this service account, so I'm pretty sure there's something deeper here - possibly not in Kudu, maybe in the IIS that sits before it.

Also, if I issue an access token w/ my service principal and pass it to the SCM endpoint as a bearer token (and not the SCM basic auth creds) - the download endpoint does work, so the creds are fine - it's something specific to the invocation of the download api from ARM.

Is there any chance someone could take a look at that ?

Here's a reproducer and an XML from the kudu logs that shows I have the contributor header set on my requests:

sub=$(az account show --query id -o tsv)
rg=appservice-reproducer
location=northeurope
app=appservice-reproducer
storage_account=appservicereproducer
sp=appservice-reproducer

az group create -n $rg -l $location
az storage account create -n $storage_account -g $rg --sku Standard_LRS
app_id=$(az functionapp create -n $app -g $rg -s $storage_account --consumption-plan-location $location --disable-app-insights --functions-version 4 --os-type Windows --runtime dotnet --query id -o tsv)
creds=$(az ad sp create-for-rbac --display-name $sp --name $sp --role Contributor --scopes $app_id)
aad_app_id=$(echo $creds | jq -r .appId)
az login -u $aad_app_id -p $(echo $creds | jq -r .password) -t $(echo $creds | jq -r .tenant) --service-principal

# sanity - should work
az rest -u "$app_id?api-version=2022-09-01"

# doesn't work
az rest -u "$app_id/functions/admin/download?api-version=2022-09-01"
Unauthorized(You do not have permission to view this directory or page.)

# also doesn't work
az rest -u "$app_id/extensions/api/functions/admin/download?api-version=2022-09-01"
Unauthorized(You do not have permission to view this directory or page.)

# but this does
az rest -u "$app_id/extensions/api/functions/admin/token?api-version=2022-09-01"

# and this does as well, albeit the invocations above should be roughly equivalent
curl https://$app.scm.azurewebsites.net/api/functions/admin/download -H "Authorization: Bearer $(az account get-access-token --resource https://management.azure.com/ --query accessToken -o tsv)"

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
PK%

# clean up
# interactive re-login
az login

az group delete -g $rg -y
az ad sp delete --id $aad_app_id
az ad app delete --id $aad_app_id

<step title="Incoming Request" date="2023-04-04T13:17:31.377" instance="5117a3" url="/api/functions/admin/token?api-version=2022-09-01" method="GET" type="request" pid="4984,2,15" Accept="*/*" Accept-Encoding="gzip, deflate" Accept-Language="en-US" Authorization="PoP..." Host="myfunc.scm.azurewebsites.net" Referer="https://management.azure.com/subscriptions/1d190b55-f402-4525-be4d-97aa349d91f3/resourceGroups/Matika-onboarding/providers/Microsoft.Web/sites/pastenwindowsconsumption/functions/admin/token?api-version=2022-09-01" User-Agent="python/3.10.10 (macOS-13.2.1-arm64-arm-64bit) AZURECLI/2.46.0 (HOMEBREW)" traceparent="00-d0ca16a4d94c3cfe8d3c23b92e48b028-1814de4f0f18c034-01" x-ms-version="20140301" x-ms-client-request-id="16cdb029-167b-44f0-a839-dcace2f960f5" CommandName="rest" ParameterSetName="-u" x-ms-arm-request-tracking-id="590569d8-caa8-405f-9162-ca1addf8c7a7" x-ms-correlation-request-id="590569d8-caa8-405f-9162-ca1addf8c7a7" x-ms-routing-request-id="QATARCENTRAL:20230404T131729Z:590569d8-caa8-405f-9162-ca1addf8c7a7" x-ms-client-location="qatarcentral" x-ms-home-tenant-id="13371337-b1ba-44c4-a73f-8928b943f202" x-ms-client-ip-address="40.122.84.241" x-ms-arm-service-request-id="13371337-61ea-402d-8846-3c5ea35ec56c" x-ms-management-group-ancestors="13371337-7cd8-9e01-2345-f6789a12345b, 13371337-b1ba-44c4-a73f-8928b943f202" x-ms-client-audience="https://management.core.windows.net/" x-ms-client-scope="" x-ms-client-acr="" x-ms-client-app-id="d993ecb4-3e54-4c0f-b1e4-01ea4651db3a" x-ms-client-app-id-acr="1" x-ms-client-tenant-id="13371337-b1ba-44c4-a73f-8928b943f202" x-ms-client-issuer="https://sts.windows.net/13371337-b1ba-44c4-a73f-8928b943f202/" x-ms-client-object-id="13371337-b7cd-4abf-a5fd-41a8def82bb4" x-ms-client-puid="" x-ms-client-alt-sec-id="" x-ms-client-principal-id="" x-ms-client-authorization-source="RoleBased" x-ms-client-identity-provider="https://sts.windows.net/13371337-b1ba-44c4-a73f-8928b943f202/" x-ms-client-principal-group-membership-source="None" x-ms-client-family-name-encoded="" x-ms-client-given-name-encoded="" x-ms-arm-network-source="PublicNetwork" x-ms-activity-vector="IN.01" x-ms-via-extensions-route="true" x-ms-geo-location="North Europe" x-ms-request-id="aeaa8c3a-e0f5-4297-a284-f1c75a8ae2b4" CLIENT-IP="13.69.227.128:12608" X-ARR-LOG-ID="aeaa8c3a-e0f5-4297-a284-f1c75a8ae2b4" X-SITE-DEPLOYMENT-ID="myfunc" WAS-DEFAULT-HOSTNAME="myfunc.scm.azurewebsites.net" X-Forwarded-Proto="https" X-AppService-Proto="https" X-Forwarded-TlsVersion="1.2" **x-ms-client-rolebased-contributor="1"** X-WAWS-Unencoded-URL="/api/functions/admin/token?api-version=2022-09-01">
<step title="Cleanup Xml Logs" date="2023-04-04T13:17:31.392"/>
<!--  duration: 31ms  -->
<step title="FunctionsController.GetAdminToken()" date="2023-04-04T13:17:31.470"/>
<!--  duration: 156ms  -->
<step title="Outgoing response" date="2023-04-04T13:17:31.689" type="response" statusCode="200" statusText="OK" Server="Microsoft-IIS/10.0" x-ms-request-id="aeaa8c3a-e0f5-4297-a284-f1c75a8ae2b4" Cache-Control="private" X-AspNet-Version="4.0.30319" Content-Type="application/json; charset=utf-8"/>
<!--  duration: 47ms  -->
</step>
<!--  duration: 375ms  -->

[FinVamp1]

@dany74q . Danny, thanks for the response and the issue you are seeing when trying to download the content via the Functions Download API are well summarized in your response above. The engineering team needs more time to evaluate why the contributor access is not honoured via the ARM control plane. Microsoft will continue to investigate this and should be able to respond back early next week.

[FinVamp1]

Hello, we have made more progress and the issue is with how the ARM Control Plane determines how to share contributor access. We are determining how best to fix this.

[dany74q]

Hey @FinVamp1 ? Any chance of an update on this ? still doesn't work, unfortunately

[dany74q]

Ping @FinVamp1, @bhagyshricompany (:

[dany74q]

Ping @FinVamp1, @bhagyshricompany <3

[dany74q]

Ping @FinVamp1, @bhagyshricompany (:

[FinVamp1]

Hello, I worked on testing this today. The fix is in a change that is rolling out all regions. In the regions where the fix has been applied I have verified that you can download the app content via ARM. Can you share which app names you have been testing in? If you access to the EUAP regions can you test there?

[dany74q]

Thanks for confirming @FinVamp1 !

Is there an ETA on when it's going to be rolled out completely by any chance ?

[dany74q]

Hey @FinVamp1, @bhagyshricompany, was this rolled out to all regions by any chance ?