microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.46k stars 2.6k forks source link

Resource should exist before deployment. Cannot read property 'getApplicationURL' of undefined. #15532

Open tomred-net opened 2 years ago

tomred-net commented 2 years ago

Note

Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo

For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

If you have an issue or request for the Azure Pipelines service, use developer community instead:

https://developercommunity.visualstudio.com/spaces/21/index.html )

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name:

list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

Environment

Issue Description

Azure Web App Deployment can't find existing Microsoft.Web/sites

[debug]Deployment Failed with Error: Error: Resource '...xxxx...' doesn't exist. Resource should exist before deployment.

This appears to be because the query is case sensitive:

[debug][GET]https://management.azure.com/subscriptions/a1dc7c64-6f29-4584-90a3-a65732d701af/resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01

Microsoft.Web%252FSites >Microsoft.Web/Sites but the correct value (and the value that is in other deployment strings is Microsoft.Web/sites note the lower case s in sites.

[Include task name(s), screenshots and any other relevant details]

Task logs

[debug]{"subscriptionID":"","subscriptionName":"","servicePrincipalClientID":"","environmentAuthorityUrl":"https://login.windows.net/","tenantID":"","url":"https://management.azure.com/","environment":"AzureCloud","scheme":"ServicePrincipal","activeDirectoryResourceID":"https://management.azure.com/","azureKeyVaultServiceEndpointResourceId":"https://vault.azure.net","azureKeyVaultDnsSuffix":"vault.azure.net","scopeLevel":"Subscription","authenticationType":"***","servicePrincipalKey":***,"isADFSEnabled":false,"applicationTokenCredentials":{"clientId":"***","domain":"***","baseUrl":"https://management.azure.com/","authorityUrl":"https://login.windows.net/","activeDirectoryResourceId":"https://management.azure.com/","isAzureStackEnvironment":false,"authType":"***","secret":***,"isADFSEnabled":false}}

Got service connection details for Azure App Service:'***'

[debug][POST]https://login.windows.net/***/oauth2/token/

[debug][GET]https://management.azure.com/subscriptions/***/resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ '***'&api-version=2016-07-01

[debug]Correlation ID from ARM api call response : b79bdf30-6fd1-4ff5-97af-955641ac01b3

[debug]Deployment Failed with Error: Error: Resource '***' doesn't exist. Resource should exist before deployment.

(node:1722) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'getApplicationURL' of undefined at ContainerWebAppDeploymentProvider. (/home/vsts/work/_tasks/AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1/4.184.4/deploymentProvider/AzureRmWebAppDeploymentProvider.js:54:73)

[debug]task result: Failed

at Generator.next (<anonymous>)

[error]Error: Resource '***' doesn't exist. Resource should exist before deployment.

[debug]Processed: ##vso[task.issue type=error;]Error: Resource 'request-to-pay-service' doesn't exist. Resource should exist before deployment.

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

[Insert error from the logs here for a quick overview]

v-ibshaik commented 2 years ago

Hello tomred-net Can you please confirm if the web site / app is already created ? The task expects the app should be pre-created.

Thanks

tomred-net commented 2 years ago

Hi, @v-ibshaik

The App Service Plan and App Services exist and can be seen when querying the az resource list. The query name matches the search name.

This is the cleaned up output:

==============================================================================
Task         : Azure App Service deploy
Description  : Deploy to Azure App Service a web, mobile, or API app using Docker, Java, .NET, .NET Core, Node.js, PHP, Python, or Ruby
Version      : 4.184.4
Author       : Microsoft Corporation
Help         : https://aka.ms/azureappservicetroubleshooting
==============================================================================
Got service connection details for Azure App Service:'request-to-pay-service'
##[error]Error: Resource 'request-to-pay-service' doesn't exist. Resource should exist before deployment.
(node:1689) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'getApplicationURL' of undefined

You will see the request that is failing is the filter resources by name and type. It is either the double escaping of the / in the Microsoft.Web%252FSites or if the query is case sensitive the use of S instead of s in sites as you can see the name matches the following result.

##[debug][GET]https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01
##[debug]Correlation ID from ARM api call response : b79bdf30-6fd1-4ff5-97af-955641ac01b3
##[debug]Deployment Failed with Error: Error: Resource 'request-to-pay-service' doesn't exist. Resource should exist before deployment.

This is the result of a az resource list command.

az resource list | grep 'request-'
    "id": "/subscriptions/.../resourceGroups/rs-core-production-australiaeast/providers/Microsoft.Web/serverFarms/request-to-pay-service-plan",
    "name": "request-to-pay-service-plan",
    "id": "/subscriptions/.../resourceGroups/rs-core-production-australiaeast/providers/Microsoft.Web/sites/request-to-pay-service",
    "name": "request-to-pay-service",
v-saikumart commented 2 years ago

Hi @tomred-net

Could you please share the details like how you are creating the resources. Is it created via pipelines before deployment or already exists?

Thank you.

tomred-net commented 2 years ago

Hi @v-saikumart,

The App Service and App Service Plan resources are created using Terraform and have been provisioned for the last week. They can be deployed to through the portal and from terraform without issue.

The serviceConnection works for other deployments such as static apps.

The deployment step for the container is as follows:

variables:
  serviceConnection: azureServiceConnection # pipelines ARM serviceConnection
  ContainerNamespace: xxx.azurecr.io   # xxx is the url for the ACR
  ContainerRepository: xxx             # request-to-pay-service
  WebAppName: xxx                      # request-to-pay-service
  ContainerTag: latest                 # a valid tag

stages:
- stage: Deploy_App_Service
  - deployment: Deploy_App_Service
    strategy:
      runOnce: 
        deploy:
          steps:
          - checkout: self
            submodules: true
          - task: AzureRMWebAppDeployment@4
            displayName: Azure App Service Deploy
            inputs:
              appType: webAppContainer
              ConnectedServiceName: $(serviceConnection) 
              WebAppName: $(WebAppName)                  
              DockerNamespace: $(ContainerNamespace)     
              DockerRepository: $(ContainerRepository)   
              DockerImageTag: $(ContainerTag)            
v-saikumart commented 2 years ago

@tomred-net

Can you share the details that you are experiencing the issue consistently or just immediately after creating app from terraform. And can you please run the mentioned GET API call from postman and confirm on same.

Thank you.

tomred-net commented 2 years ago

HI @v-saikumart,

I am experiencing this consistently.

The query from the request fails:

https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01 

{
    "value": []
}

When I fix the URL encoding it passes

https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web/Sites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01

{
    "value": [
        {
            "id": "/.../request-to-pay-service",
            "name": "request-to-pay-service",
            "type": "Microsoft.Web/sites",
            "kind": "app,linux,container",
            "location": "australiaeast",
            "identity": {
                "principalId": "...",
                "tenantId": "...",
                "type": "SystemAssigned"
            },
            "tags": {}
        }
    ]
}
v-saikumart commented 2 years ago

Hi @tomred-net
Could you please try https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01 instead of https://management.azure.com/subscriptions/.../resources?$filter=resourceType EQ 'Microsoft.Web/Sites' AND name EQ 'request-to-pay-service'&api-version=2016-07-01 and confirm on same.

Thank you.

tomred-net commented 2 years ago

@v-saikumart as demonstrated in my previous reply I tried the first url (containing 'Microsoft.Web%252FSites' ) which is the same as the value submitted by the azure task and the result is an empty array.

{
    "value": []
}

If I replace the double url encoding of '/' then I get the successful results as shown in my previous answer.

To clarify / is encoded to %2f which appears to be re-encoded to %252f as original % is encoded as %25. url_encode('/') = %2f url_encode( url_encode('/') ) = %252f

tomred-net commented 2 years ago

@v-saikumart @v-ibshaik is there anything else I can do to help you debug this issue?

v-saikumart commented 2 years ago

@tomred-net Could you please share the below mentioned details, as we are try to reproduce the issue and it is working as expected.

i. Complete Pipeline logs. ii. Complete YAML file.

Thank you.

RSteeno commented 2 years ago

@tomred-net

Were you able to find a fix for this? I am experiencing a similar issue, but it is sporadic.

  1. Provision app service with Terraform
  2. Wait 5 minutes (previously I had a 60 second wait and it was working fine for weeks).
  3. Deploy web site via AzureWebApp@1 task

Up until yesterday this was working and now sometimes it works and other times it fails.

Error:

Got service connection details for Azure App Service:'app-xxxx-f48362-eaus'
##[error]Error: Resource 'app-xxxx-f48362-eaus' doesn't exist. Resource should exist before deployment.
cswierczynski commented 2 years ago

@tomred-net

Were you able to find a fix for this? I am experiencing a similar issue, but it is sporadic.

  1. Provision app service with Terraform
  2. Wait 5 minutes (previously I had a 60 second wait and it was working fine for weeks).
  3. Deploy web site via AzureWebApp@1 task

Up until yesterday this was working and now sometimes it works and other times it fails.

Error:

Got service connection details for Azure App Service:'app-xxxx-f48362-eaus'
##[error]Error: Resource 'app-xxxx-f48362-eaus' doesn't exist. Resource should exist before deployment.

I am also seeing the exact message. Was working up until today

ikhoury-nlo commented 2 years ago

@tomred-net

Were you able to find a fix for this? I am experiencing a similar issue, but it is sporadic.

  1. Provision app service with Terraform
  2. Wait 5 minutes (previously I had a 60 second wait and it was working fine for weeks).
  3. Deploy web site via AzureWebApp@1 task

Up until yesterday this was working and now sometimes it works and other times it fails.

Error:

Got service connection details for Azure App Service:'app-xxxx-f48362-eaus'
##[error]Error: Resource 'app-xxxx-f48362-eaus' doesn't exist. Resource should exist before deployment.

I've been trying to deploy my function app all day, exact same steps and error at the end. Using the AzureFunctionApp@1 task in a YAML pipeline. The app service plan perfectly exists in the azure portal, but not in devops. It also does not appear in the drop down when trying to create a classic release task. The service connection is owner in the target subscription, no reason for the function app "not to exist".

ikhoury-nlo commented 2 years ago

Just started working today! I hope it's a permanent fix...

tomred-net commented 2 years ago

@tomred-net Could you please share the below mentioned details, as we are try to reproduce the issue and it is working as expected.

i. Complete Pipeline logs. ii. Complete YAML file.

Thank you.

Hi @v-saikumart,

I am sorry I did not receive a notification of you edits to your original reply, hence I did not notice the request for more information. I will run the deployment now and extract the logs. It is still failing for the same reason.

Dermot

tomred-net commented 2 years ago

@v-saikumart

Please find attached both the log file and the azure-pipeline.yml

Dermot azure-pipeline.yml.txt failed.deployment.txt

tomred-net commented 2 years ago

@RSteeno No fix so far. It is still happening for us.

v-saikumart commented 2 years ago

@tomred-net Thank you for the response. We will investigate further and share the status accordingly. Thank you.

jschiessl commented 2 years ago

Same problem for me and my team of 15 devs, this is heavily impacting us currently. The only difference to the behavior reported by @tomred-net is that is does not fail always, but very often. This is even more annoying.

RSteeno commented 2 years ago

I have an case open with MSFT. The engineer referenced an ongoing Resource Manager issue dated Jan 13th which could be related.

https://status.azure.com/en-us/status/history/

v-saikumart commented 2 years ago

@tomred-net Could please verify the SPN role and confirm the web app(request-to-pay-service) is present in that subscription(PAYGO Subscription). Thank you.

mdubdev commented 2 years ago

I too am dealing with this issue and have a support case open with MS. Please let me know if there is anything I can provide to assist in resolving this issue as it is causing a lot of issues for our devs. I also want to point out that this is happening with our function deployments as well.

tomred-net commented 2 years ago

@tomred-net Could please verify the SPN role and confirm the web app(request-to-pay-service) is present in that subscription(PAYGO Subscription). Thank you.

Hi @v-saikumart, the request-to-pay-service is in the PAYGO Subscription. The Service Principle has Azure Service Management API permissions and is a 'Cloud application administrator'. The SPN is capable of deploying Static Web Apps to Microsoft.Web%2FStaticSites and Containers to Microsoft.ContainerRegistry%2Fregistries without issue.

lmrisdal commented 2 years ago

Also dealing with this issue. Seems completely random if it fails or not. Deployed 4 apps just now. 3 of there were successful, but the last one refuses to deploy.

lkfabu2022 commented 2 years ago

Having the same issue. see https://developercommunity.visualstudio.com/t/Error:-Resource-DiagnosisCodeApi-doesn/1646720?viewtype=all&space=21&ftype=problem&sort=newest

lkfabu2022 commented 2 years ago

@v-saikumart as demonstrated in my previous reply I tried the first url (containing 'Microsoft.Web%252FSites' ) which is the same as the value submitted by the azure task and the result is an empty array.

{
    "value": []
}

If I replace the double url encoding of '/' then I get the successful results as shown in my previous answer.

To clarify / is encoded to %2f which appears to be re-encoded to %252f as original % is encoded as %25. url_encode('/') = %2f url_encode( url_encode('/') ) = %252f

I have old resources working fine and resources cloned from an existing app service also working. Issue is just w/ newly created resources. However, both have the same encoding.

test-02 is cloned and working

[GET]https://management.azure.com/subscriptions/xxxxxxxxxxxxxxxx/resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'app-xxxxxxx-testing-02'&api-version=2016-07-01

testing01 is new and broken

[GET]https://management.azure.com/subscriptions/xxxxxxxxxxxxxxxxx/resources?$filter=resourceType EQ 'Microsoft.Web%252FSites' AND name EQ 'app-xxxxxxxxx-testing01'&api-version=2016-07-01

joshuasmartin commented 2 years ago

I am experiencing this same error with new Azure Functions. 1 out of 4 of them showed up in the drop down for the existing service connection and even a new service connection.

adalfa commented 2 years ago

I can confirm the same problem on a newly created resource, it fails very very often

Prem-Codes commented 2 years ago

My team have been facing the same issue. It fails sporadically.

joshuasmartin commented 2 years ago

All of my Azure Functions have now showed up this morning that were created yesterday. They were not available as of 10:30-11pm last night after being created 4-5 hours previously.

wkoeter commented 2 years ago

We also have this issue now. Out of the 5 deploys I just did to the same Azure Function, only 3 were successfull. 2 of them failed with (obfuscated function name):

Tried with task AzureRmWebAppDeployment@4 and task AzureFunctionApp@1

##[error]Error: Resource '********' doesn't exist. Resource should exist before deployment.

v-saikumart commented 2 years ago

@tomred-net We are unable to repro the issue. Is still issue exist, please confirm.

Thank you.

lkfabu2022 commented 2 years ago

Just tested. My new resource created on the 25th showed up too, however anything created today isn't showing up. Clones aren't either showing up in the task GUI selection, this was working yesterday. They are working when running the task.

Please, please get this fixed ASAP. We're seriously impeded by this issue.

Test Access, running Get-AzWebApp | Where-Object {$_.name -like "*test*"} | select -Property Name,ResourceGroup,LastModifiedTimeUtc in the task, with the same service account. 2022-01-27_12h14_37

Error 2022-01-27_12h15_52

borondy commented 2 years ago

We are facing similar issue as well. Basically what happens behind the curtains is that deployment tasks are calling the https://management.azure.com/subscriptions/{subscription-id}/resources API to validate the target resources of the deployments. Nowadays the result of this call does not contain every resource for some reason, if the target resource is missing, the deployment fails.

metaxasa commented 2 years ago

Just to expand upon @borondy's comment. Looks like it could be the PreDeploymentStep() which tries to get the RG name:

https://github.com/microsoft/azure-pipelines-tasks/blob/f66f0746753fcc197d1a5eb26a13eb87813416bd/Tasks/AzureRmWebAppDeploymentV4/deploymentProvider/AzureRmWebAppDeploymentProvider.ts#L36

AzureResourceFilterUtility.getResourceGroupName() tries to get the web app (Microsoft.Web/Sites) resource:

https://github.com/microsoft/azure-pipelines-tasks/blob/f66f0746753fcc197d1a5eb26a13eb87813416bd/Tasks/AzureRmWebAppDeploymentV4/operations/AzureResourceFilterUtility.ts#L8

Using Resources.getResources() which grabs all resources for the given subscription

https://github.com/microsoft/azure-pipelines-tasks/blob/f66f0746753fcc197d1a5eb26a13eb87813416bd/common-npm-packages/azure-arm-rest-v2/azure-arm-resource.ts#L44

That match this filter:

https://github.com/microsoft/azure-pipelines-tasks/blob/f66f0746753fcc197d1a5eb26a13eb87813416bd/common-npm-packages/azure-arm-rest-v2/azure-arm-resource.ts#L45

And then throws if it doesn't find it.

https://github.com/microsoft/azure-pipelines-tasks/blob/f66f0746753fcc197d1a5eb26a13eb87813416bd/Tasks/AzureRmWebAppDeploymentV4/operations/AzureResourceFilterUtility.ts#L11

lmrisdal commented 2 years ago

I have also recently experienced that commands such as "az resource list" not always gives correct results. E.g. a cosmos db account would should up in the Azure Portal and with "az cosmosdb list", but not with "az resource list".

Not a clue if there is a connection, but I figured it might be interesting.

adalfa commented 2 years ago

Following yesterday deployment the pipeline seems to work fine today : does anyone experiencing only in new resources? Also does it occurs on resources that are deployed from an ARM template? It seems to me that the resource manager api has some consistency issue

safari137 commented 2 years ago

This is really kind of bizarre. I'm having the same issue, but it seem to only happen when I use an ARM template.

I was able to work around the issue initially by creating the resource (function app) through the portal. I was then able to deploy to it fine from Azure Devops.

However, after deploying the ARM template (incremental mode), it stopped working right after. I was hoping to still use the ARM template so that I wouldn't have to manually configure everything else about the app.

cl-zacrollins commented 2 years ago

I'm seeing same issue as everyone above. My pipeline uses a bicep/arm template to deploy new app services and then deploy my application code to these newly created app services. This worked flawlessly until a couple weeks ago. Now I get ##[error]Error: Resource '{MyAppServiceName}' doesn't exist. Resource should exist before deployment. just about every time I run the deploy pipeline. I found if I wait 10-15 minutes after the deploy fails and "Rerun Failed Jobs" the second attempt will complete successfully. Unfortunately, I HAVE to wait that 10-15 minutes, so adding some custom retry logic or using the "Automatic retries for task" with "retryCountOnTaskFailure" wont work for me. This is really becoming a headache. Hopefully we get some help on this soon!

0xUnicorn commented 2 years ago

I have this issue aswell, when creating a WebApp through the Portal GUI. I'm able to select my WebApp through the release pipeline task when selecting the deployment resource. Resources has existed for more than an hour.

maikelvanhaaren commented 2 years ago

Same issue for me. Tried multiple things but none works out. I've created a duplicated ticket which is now closed: #15806. Hope this issue gets fixed soon because it's blocking our development flow.

maikelg commented 2 years ago

I'm running into the same issue. My deployment process has been working without issue until a couple of weeks ago. Now it regularly fails with this error 'Resource should exist before deployment'.

Jacob-Steentoft commented 2 years ago

I'm also running into the 'Resource should exist before deployment' error for 2 different tenants and subscriptions, both my personal, and for work. Are there any workarounds to neatly rerun the failed job from the deployment? This is causing issues for our deployments.

NickRhead commented 2 years ago

Same issue as everyone else. Created resources via Terraform and still after 3 hours I can't deploy code to those resources. Used to work with the same resources now it doesn't

My resources are in west europe also

lkfabu2022 commented 2 years ago

FYI I didn't check again recently, but my initial issue was with resources created manually in the GUI.

adalfa commented 2 years ago

Hello to all, since the deployment (by template or portal) does not seems to matter, could it be a regional issue? all the resources affected were for us in West Europe

maartenvandiemen commented 2 years ago

Hello to all, since the deployment (by template or portal) does not seems to matter, could it be a regional issue? all the resources affected were for us in West Europe

I just deployed a WebApp in West Europe en was facing the issues as described here (9 out of 10 deployments failed). Deploying the same app to Central US and no issue at all.

cl-zacrollins commented 2 years ago

@adalfa all my deployments are in the West US 2 region.

v-saikumart commented 2 years ago

@here We are investigating on the issue, we will provide status on same at earliest.

Thank you

maikelvanhaaren commented 2 years ago

@v-saikumart If you need more information for understanding this issue please reach out.