owenmather commented 5 years ago

Description az webapp config ssl bind command fails with: certificate "xyz" not found when pfx cert is uploaded to webapp in different resource group than app service plan

To Reproduce Create App Service Plan in resource group 'A' Create WebApp 'myApp' on Plan in new resource group 'B' Upload .pfx cert to WebApp

az webapp config ssl list --resource-group 'B'

This shows certificate with thumbprint 'xyz' found

az webapp config ssl bind --certificate-thumbprint 'xyx' --ssl-type SNI --name 'myApp' --resource-group 'B'

Results in Error certificate 'xyz' not found

--debug shows az webapp config ssl bind is searching for pfx cert in resource group 'A' where the AppService plan is located.

Capture

.pfx cert is actually located in resource group 'B' alongside WebApp.

Expected behavior Command should run successfully.

Uploaded private key certs are located in the rg of the WebApp they are initially uploaded to. Uploaded private key certs can be shared across all WebApps in the same App Service Plan regardless of resource group.

Suggested Fix: az webapp config ssl bind should search for a certificate with matching a thumbprint in all resource groups containing WebApps of the parent App Service Plan to the bind target.

Workaround Uploading the .pfx cert to a WebApp in the same resource group as the App Service Plan works as expected.

panchagnula commented 5 years ago

@owenmather what is the version of CLI you are using? please run az --version & share the details here? thank you.

owenmather commented 5 years ago

@panchagnula Seeing this with Hosted Ubuntu 1604 on Azure DevOPs and on Azure portal with Powershell Cloud Shell:

Hosted Ubuntu 1604

2019-07-18T09:30:21.8786881Z [command]/usr/bin/az --version 2019-07-18T09:30:23.7790310Z azure-cli 2.0.67 2019-07-18T09:30:23.7791069Z 2019-07-18T09:30:23.7791400Z acr 2.2.9 2019-07-18T09:30:23.7791791Z acs 2.4.4 2019-07-18T09:30:23.7792174Z advisor 2.0.1 2019-07-18T09:30:23.7792394Z ams 0.4.7 2019-07-18T09:30:23.7792658Z appservice 0.2.21 2019-07-18T09:30:23.7792977Z backup 1.2.5 2019-07-18T09:30:23.7793499Z batch 4.0.3 2019-07-18T09:30:23.7793791Z batchai 0.4.10 2019-07-18T09:30:23.7794023Z billing 0.2.2 2019-07-18T09:30:23.7794298Z botservice 0.2.2 2019-07-18T09:30:23.7794567Z cdn 0.2.4 2019-07-18T09:30:23.7794841Z cloud 2.1.1 2019-07-18T09:30:23.7795071Z cognitiveservices 0.2.6 2019-07-18T09:30:23.7795705Z command-modules-nspkg 2.0.2 2019-07-18T09:30:23.7796070Z configure 2.0.24 2019-07-18T09:30:23.7796348Z consumption 0.4.4 2019-07-18T09:30:23.7796612Z container 0.3.18 2019-07-18T09:30:23.7796890Z core 2.0.67 2019-07-18T09:30:23.7797165Z cosmosdb 0.2.11 2019-07-18T09:30:23.7797435Z deploymentmanager 0.1.1 2019-07-18T09:30:23.7797701Z dla 0.2.6 2019-07-18T09:30:23.7797982Z dls 0.1.10 2019-07-18T09:30:23.7798322Z dms 0.1.4 2019-07-18T09:30:23.7798589Z eventgrid 0.2.4 2019-07-18T09:30:23.7798862Z eventhubs 0.3.7 2019-07-18T09:30:23.7799093Z extension 0.2.5 2019-07-18T09:30:23.7799365Z feedback 2.2.1 2019-07-18T09:30:23.7799630Z find 0.3.4 2019-07-18T09:30:23.7799904Z hdinsight 0.3.5 2019-07-18T09:30:23.7800134Z interactive 0.4.5 2019-07-18T09:30:23.7800407Z iot 0.3.11 2019-07-18T09:30:23.7800712Z iotcentral 0.1.7 2019-07-18T09:30:23.7800975Z keyvault 2.2.16 2019-07-18T09:30:23.7801205Z kusto 0.2.3 2019-07-18T09:30:23.7801838Z lab 0.1.8 2019-07-18T09:30:23.7802113Z maps 0.3.5 2019-07-18T09:30:23.7802364Z monitor 0.2.15 2019-07-18T09:30:23.7802597Z natgateway 0.1.1 2019-07-18T09:30:23.7802872Z network 2.5.2 2019-07-18T09:30:23.7803147Z nspkg 3.0.3 2019-07-18T09:30:23.7803407Z policyinsights 0.1.4 2019-07-18T09:30:23.7803682Z privatedns 1.0.2 2019-07-18T09:30:23.7803912Z profile 2.1.5 2019-07-18T09:30:23.7804218Z rdbms 0.3.12 2019-07-18T09:30:23.7804519Z redis 0.4.4 2019-07-18T09:30:23.7804798Z relay 0.1.5 2019-07-18T09:30:23.7805253Z reservations 0.4.3 2019-07-18T09:30:23.7805535Z resource 2.1.16 2019-07-18T09:30:23.7805814Z role 2.6.4 2019-07-18T09:30:23.7806096Z search 0.1.2 2019-07-18T09:30:23.7806330Z security 0.1.2 2019-07-18T09:30:23.7806609Z servicebus 0.3.6 2019-07-18T09:30:23.7807000Z servicefabric 0.1.20 2019-07-18T09:30:23.7807261Z signalr 1.0.1 2019-07-18T09:30:23.7807484Z sql 2.2.5 2019-07-18T09:30:23.7807738Z sqlvm 0.2.0 2019-07-18T09:30:23.7807991Z storage 2.4.3 2019-07-18T09:30:23.7808439Z telemetry 1.0.2 * 2019-07-18T09:30:23.7808712Z vm 2.2.23 2019-07-18T09:30:23.7808897Z 2019-07-18T09:30:23.7809482Z Python location '/opt/az/bin/python3' 2019-07-18T09:30:23.7810098Z Extensions directory '/home/vsts/.azure/cliextensions'

Powershell Cloud Shell

PS Azure:> az --version azure-cli 2.0.69

command-modules-nspkg 2.0.3 core 2.0.69 nspkg 3.0.4 telemetry 1.0.3

Extensions: application-insights 0.1.1

Python location '/opt/az/bin/python3' Extensions directory '/home/owen/.azure/cliextensions'

Python (Linux) 3.6.5 (default, Jul 11 2019, 08:40:03) [GCC 5.4.0 20160609]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

panchagnula commented 5 years ago

@owenmather thanks for the info. I tried the different RG scenario out & was not able to repro this. Can you run the command with --debug & share where exactly the NoFound Error is coming from. You can remove any sensitive info before you share here or you can send me an email with the details directly at sisirap@microsoft.com. thank you.

owenmather commented 5 years ago

@panchagnula

Sure please find log file with debug attached from running below commands:

az webapp config ssl list --resource-group 'BB' --debug
#Shows thumbprint 'A00668B05EE8FE61DCDFB3B123FD2BC24E5AF8B0" found

az webapp config ssl bind --certificate-thumbprint "A00668B05EE8FE61DCDFB3B123FD2BC24E5AF8B0" --ssl-type SNI --name 'myWebAppInRgB' --resource-group 'BB' --debug

#Gives error certificate for thumbprint 'A00668B05EE8FE61DCDFB3B123FD2BC24E5AF8B0' not found.

azureSSLBindings.log

panchagnula commented 5 years ago

thanks for the info will look into this.

panchagnula commented 5 years ago

Still unable to repro this on our end - moving to next sprint to investigate further.

owenmather commented 5 years ago

@panchagnula

I have set up a repo so you can reproduce the error using a test pfx cert. https://github.com/owenmather/az-webapp-bind-err-demo

Running the script from here reproduces the error for me every time:

Certificate for thumbprint '3200E6FD65363F3611CDF63DF2A14B0FB0DD4560' not found. Hopefully this will help you recreate this issue.

The error looks like it comes from here: https://github.com/Azure/azure-cli/blob/9d9c2a97dc178a0eccd08e56b6e66a690c9075f7/src/azure-cli/azure/cli/command_modules/appservice/custom.py#L1890-L1893

This command is only searching the resource group of the app service plan for certs.

We can see through the portal by enabling hidden types that the cert gets stored in the same location as the Web App and not the App Service plan Capture

When I update the custom.py file to search the all resource groups of the child webapps for a matching cert and test then it works correctly.

panchagnula commented 5 years ago

in your command az webapp config ssl bind what is the RG value you are using - the ASP one or the Webapp one?

owenmather commented 5 years ago

@panchagnula

what is the RG value you are using - the ASP one or the Webapp one?

The Webapp one.

Using the Asp RG would produce the following error: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Web/sites/MyWebapp2139102312' under resource group 'AspRG' was not found."}}

You can see the full script I am using below:

MyWebapp=MyWebapp2139102312
echo "Creating resource groups. . ."
echo
az group create -l northeurope -n 'AspRG' --output none
az group create -l northeurope -n 'WebappRG' --output none

echo  "Creating app service plan MyASP. . ."
planId=$(az appservice plan create -n "MyASP" -g "AspRG" --sku "S1" -l northeurope --query id)
echo  "Created with id: $planId"
echo

echo  "Creating webapp $MyWebapp. . ."
webappId=$(eval `echo az webapp create -g WebappRG -p $planId -n  $MyWebapp --query id`)
echo  "Created webapp with id: $webappId \n"
echo

echo "Uploading pfx cert. . ."
az webapp config ssl upload --certificate-file testcert.pfx --certificate-password "test1234" --name "$MyWebapp" -g "WebappRG" --output none
echo
echo "Retrieving thumbprint. . ."
thumbprint=$(az webapp config ssl upload \
    --name "$MyWebapp" \
    --resource-group WebappRG \
    --certificate-file testcert.pfx \
    --certificate-password test1234 \
    --query thumbprint \
    --output tsv)

echo  "found cert with thumbprint: $thumbprint"
echo
echo "Attempting to bind webapp to pfx cert in seperate rg to asp.."
echo
az webapp config ssl bind --certificate-thumbprint $thumbprint --ssl-type SNI --name "$MyWebapp" -g "WebappRG"

Repo I have linked above includes the throwaway cert used https://github.com/owenmather/az-webapp-bind-err-demo

phekmat commented 5 years ago

When I update the custom.py file to search the all resource groups of the child webapps for a matching cert and test then it works correctly.

This approach should also fix the issue I had in https://github.com/Azure/azure-cli/issues/8157#issuecomment-490180966 @panchagnula (assuming it wasn't resolved in the meantime)

malingel commented 5 years ago

When I update the custom.py file to search the all resource groups of the child webapps for a matching cert and test then it works correctly.

This approach should also fix the issue I had in #8157 (comment) @panchagnula (assuming it wasn't resolved in the meantime)

I think this should go even deeper. You can technically upload a wild card cert to a web app and it would be placed in that web app's RG. If you try and upload the same cert to a different web app in a different RG on a different service plan, you would get a conflict that a cert with that thumbprint already exists.

This binding should follow the same rules that the upload does, which seems to be unique to the subscription, not unique to the web app, service plan, or resource group. This seems to be what rules the portal UI follow as well.

yonzhan commented 5 years ago

@panchagnula What is the current status of this issue ?

panchagnula commented 5 years ago

we need to work with the feature crew to fix the API to handle these scenarios we don't want to make these changes on the client to support cross RG. No ETA at this point

panchagnula commented 5 years ago

@owenmather sorry for delay on this - for SSL upload operation the RG you pass to the command should be the RG of the Appservice plan & not the app - that will make sure that the cert gets uploaded to the right location that is accessible for the bind operation

This is a common issue when the upload is done directly via ARM deployment or PowerShell or CLI where the RG provided is the RG of the app & not the ASP. The API is making a change handle this correctly after which CLI & PowerShell command will updated as well until then the workaround would be to pass the RG of the ASP & not the webapp when running SSL commands.

dombarnes commented 4 years ago

Its very confusing (and seemingly undocumented) that you need to specify the resource group for the app service plan rather than the web app, which is the domain you're running the command against. It would be useful to also/at least get https://docs.microsoft.com/en-us/azure/app-service/scripts/cli-configure-ssl-certificate and https://docs.microsoft.com/en-us/cli/azure/webapp/config/ssl?view=azure-cli-latest#az-webapp-config-ssl-bind amended with a note

dombarnes commented 4 years ago

Having just tried this, I've actually received an error {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Web/sites/webapp-name' under resource group 'ASP_group_name' was not found."}} Guess I may just have to reorganise some resource groups

sanderaernouts commented 4 years ago

@dombarnes just tried this with 2.0.77 of the Azure CLI and it works. The command did fail before updating to the latest version of the CLI

dombarnes commented 4 years ago

@sanderaernouts an I'm still on 2.0.70 so I'll get updated! Thank you :)

yonzhan commented 4 years ago

How about this issue?

panchagnula commented 4 years ago

closing this based on comments that 2.0.77 works from above plus we have plans to update SSL commands& work on making help better as a part of those changes. Thanks!

collinstevens commented 4 years ago

@panchagnula @yonzhan I am still experiencing this issue on version 2.0.77.

I have an App Service Environment in rg-general. I have an App Service Plan in rg-general in the App Service Environment. I have an App Service in rg-app named app-myApp in the App Service Plan.

I have uploaded a certificate to rg-general and az webapp config ssl list -g rg-general displays the certificate with a thumbprint of F1B33D637F075385ED40DF3D31F9C6CE2D545009

Attempting to run az webapp config ssl bind --certificate-thumbprint F1B33D637F075385ED40DF3D31F9C6CE2D545009 --ssl-type SNI -g rg-app -n app-myApp returns the error: Certificate F1B33D637F075385ED40DF3D31F9C6CE2D545009 was not found.

I have also tried to upload the certificate to the resource group of the App Service Plan, but it returns the same result.

I have also updated to version 2.0.80 (the latest at time of writing) and I am receiving the same error.

panchagnula commented 4 years ago

@collinstevens how did you upload the certificate with this thumbprint?

collinstevens commented 4 years ago

@panchagnula I am executing an ARM template generated by "Import Key Vault Certificate" under "Private Key Certificates (.pfx)" in the "TLS/SSL Settings" blade. I generated the same certificate as was generated by the Portal option, yet it is unable to find the thumbprint when created through the ARM template.

job-lukasz commented 4 years ago

What is status of this issue. As I can see it is close however in version:

{
  "azure-cli": "2.4.0",
  "azure-cli-command-modules-nspkg": "2.0.3",
  "azure-cli-core": "2.4.0",
  "azure-cli-nspkg": "3.0.4",
  "azure-cli-telemetry": "1.0.4",
  "extensions": {}
}

It still does not work.

My scenario according to your comment:	type	name	resource group
App service plan	ASP1	RG1
Web app	WEBAPP2	RG2

thumbprint=$(az webapp config ssl upload --certificate-file MY_DOMAIN.pfx --certificate-password test1234 --name WEBAPP2 --resource-group RG2 -o tsv --query thumbprint)
az webapp config ssl bind --certificate-thumbprint ${thumbprint} --resource-group RG1 --name WEBAPP2 --ssl-type SNI

Its fail with comment:

{"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Web/sites/WEBAPP2' under resource group 'RG1' was not found."}}

When I try to execute:

az webapp config ssl bind --certificate-thumbprint ${thumbprint} --resource-group RG2 --name WEBAPP2 --ssl-type SNI

then error:

Certificate for thumbprint 'XXXXXXCERTTHUMBPRINTXXXXX' not found.

Also upload certificate to app service plan:

az webapp config ssl upload --certificate-file MY_DOMAIN.pfx --certificate-password test1234 --name ASP1 --resource-group RG1 -o tsv --query thumbprint

fails with error:

{"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Web/sites/ASP1' under resource group 'RG1' was not found."}}

Is there any progres to resolve this issue?

mattou07 commented 4 years ago

Hi I am also getting this issue, As of writing this I am using the latest version of the cli with powershell 7

PS C:\Users\matth> az --version
azure-cli                          2.5.1

command-modules-nspkg              2.0.3
core                               2.5.1
nspkg                              3.0.4
telemetry                          1.0.4

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\matth\.azure\cliextensions'

Python (Windows) 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 02:47:15) [MSC v.1900 32 bit (Intel)]

I manually added the certificate via the portal and I am now trying to reference using the thumbprint I obtained from the portal.

I am running the following command: az webapp config ssl bind --certificate-thumbprint $thumbprint --ssl-type SNI --name $app --resource-group $rg --subscription $sub

If I use the ASP resource group I get this error, I have edited the names: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Web/sites/my-web-app' under resource group 'my-web-app-resource-group' was not found."}}

If I use the Web App resource group I get this: Certificate for thumbprint 'the thumbprint' not found.

AdamCoulterOz commented 4 years ago

this is still an issue as a new version of it is #13929

AdamCoulterOz commented 4 years ago

@panchagnula

@owenmather sorry for delay on this - for SSL upload operation the RG you pass to the command should be the RG of the Appservice plan & not the app - that will make sure that the cert gets uploaded to the right location that is accessible for the bind operation

This is a common issue when the upload is done directly via ARM deployment or PowerShell or CLI where the RG provided is the RG of the app & not the ASP. The API is making a change handle this correctly after which CLI & PowerShell command will updated as well until then the workaround would be to pass the RG of the ASP & not the webapp when running SSL commands.

I disagree with this... all the docs imply the certificate upload and/or a managed certificate are against the web app, not the service plan. In fact when you do it via the portal, it creates them in the RG for the web app, not the service plan. I've even inspected the REST API calls made from the portal to confirm this.

What you are suggesting is a possible work around (although I haven't tested it) it cant work for me, as the scope of the app service has different permissions than the web app, as the app service plan is shared.

owenmather commented 4 years ago

The suggested fix does not work. This was never fixed. Not sure why it was ever closed.

If you try to upload the SSL cert to the App Service the below happened:

az webapp config ssl upload --certificate-file MY_DOMAIN.pfx --certificate-password test1234 --name ASP1 --resource-group RG1 -o tsv --query thumbprint fails with error:

{"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Web/sites/ASP1' under resource group 'RG1' was not found."}}

Kentrg11 commented 4 years ago

Why is this closed

genscape-agodfrey commented 3 years ago

No seriously why is this closed

sanderaernouts commented 3 years ago

Not sure if this will work, but I had a similar issue with deploying a functions app where the app service plan was in a different resource group the other day. I had to use the resource ID instead of the name. You can try using the --ids parameter to pass the web app resource id instead of --name ... according to the help docs of the CLI. Not sure if --resource-groups is required, when you use --ids since it seems to refer to the app service plan resource group.

> az webapp config ssl bind --help
....
--ids                               : One or more resource IDs (space-delimited). It should be a
                                          complete resource ID containing all information of
                                          'Resource Id' arguments. You should provide either --ids
                                          or other 'Resource Id' arguments.
....

MarkPerry24 commented 3 years ago

This shouldn't be closed as same problem exists in: az version { "azure-cli": "2.17.1", "azure-cli-core": "2.17.1", "azure-cli-telemetry": "1.0.6", "extensions": { "ai-examples": "0.2.5", "azure-cli-ml": "1.19.0" } }

I have found a workaround for this which is to use the generic "az resource" which allows you to jump in and do things via the API directly. This is not an API bug as this works very basic for illustration only:

az webapp config hostname add {yourAppDomainToBind e.g. mydomain.com} az webapp config ssl upload {certUploadParams} az resource update --ids "{WebAppID}/hostNameBindings/{yourAppDomainToBind e.g. mydomain.com}?api-version=2019-08-01" --set properties.thumbprint={certThumprint} --set properties.sslState=SniEnabled

You need to pass both --set params or nothing will be set e.g. you cannot just send certThumprint only hoping to see it updated but not active for test though you could set it to Disabled explicitly. Replace SniEnabled with IPBasedEnabled or Disabled as suits need as per: https://docs.microsoft.com/en-us/rest/api/appservice/webapps/createorupdatehostnamebinding

rafalszu commented 3 years ago

Agreed, this shouldn't be closed, the bug is still there - ran recently into a very similar issue. The actual app certificate is in resource group A, app service plan and app services are in resource group B. While trying to bind the cert, I'm getting a random 404 "certificate was not found" and sometimes everything goes ok. Azure portal seems to be affected as well, there are times when I can't add a ssl binding for my custom domain because No certificates match the selected custom domain. which is not true.

I think this started occurring after moving the app service plan to linux, it was ok on Windows.

coderpatros commented 3 years ago

Thank you @MarkPerry24. You just saved me updating a whole bunch of web apps manually.

giovannifl commented 3 years ago

Any updates on this topic ?

4sudiptodas commented 3 years ago

This shouldn't be closed as same problem exists in: az version { "azure-cli": "2.17.1", "azure-cli-core": "2.17.1", "azure-cli-telemetry": "1.0.6", "extensions": { "ai-examples": "0.2.5", "azure-cli-ml": "1.19.0" } }

I have found a workaround for this which is to use the generic "az resource" which allows you to jump in and do things via the API directly. This is not an API bug as this works very basic for illustration only:

az webapp config hostname add {yourAppDomainToBind e.g. mydomain.com} az webapp config ssl upload {certUploadParams} az resource update --ids "{WebAppID}/hostNameBindings/{yourAppDomainToBind e.g. mydomain.com}?api-version=2019-08-01" --set properties.thumbprint={certThumprint} --set properties.sslState=SniEnabled

You need to pass both --set params or nothing will be set e.g. you cannot just send certThumprint only hoping to see it updated but not active for test though you could set it to Disabled explicitly. Replace SniEnabled with IPBasedEnabled or Disabled as suits need as per: https://docs.microsoft.com/en-us/rest/api/appservice/webapps/createorupdatehostnamebinding

to get the --ids use $rlist = az webapp config hostname list --resource-group $webResourceGroup --webapp-name $webappName | ConvertFrom-Json

ashxos commented 2 years ago

As this case is closed and I also have faced same issue with the latest version of az cli, I have created another issue for this. https://github.com/Azure/azure-cli/issues/21133

Azure / azure-cli

az webapp config ssl bind results in error when webapp is in different rg than app service plan #9972

This shows certificate with thumbprint 'xyz' found

Results in Error certificate 'xyz' not found

Hosted Ubuntu 1604

Powershell Cloud Shell