pipeline failing with An error occurred while provisioning resources (Error Type: Failure). The remote provider was unable to process the request.

[somiya9868]

Description
The azure pipeline is failing with error

"An error occurred while provisioning resources (Error Type: Failure). The remote provider was unable to process the request."  

Area for Triage:

Question, Bug, or Feature?:
Bug

Virtual environments affected

• [ ] Image: ubuntu-latest

Expected behavior
The pipeline should have executed with no issues.

Actual behavior
Suddenly few pipelines stopped running with the mentioned error.

Repro steps
A description with steps to reproduce the issue. If your have a public example or repo to share, please provide the link.

1. ran the pipeline with Image: ubuntu-latest

[suboorjamalredmane]

It is resolved in Central US now.

[MateusMo]

Thank you Bill Gates, It's working here in Brazil

[wggley]

Can someone please point me out where we can create tickets for M$?

As everyone here figured we couldn't find a place to report and googled here.

+1 Brazil working again

[ajsharmaofficial]

Seems this is not fully fixed yet. Started seeing it again.

[Qi-Jian-Huang-DevOps]

@ajsharmaofficial canada central is working fine.

[ajsharmaofficial]

Getting this error on canada central. is there something else could cause the same error. I could be missing something.

[ajsharmaofficial]

Just changed to a self hosted agent and its working fine. There might be something wrong on Canada central Microsoft hosted agent.

[Qi-Jian-Huang-DevOps]

I am in canada central, and it is working fine for me, I just tested it. I am using windows-latest hosted agent.

[andrewwiebe]

This was corrected last week, however it just started happening again this morning. Location CANADA, build agent linux latest.

[andrewwiebe]

I just restarted the failed build and its fine now. Maybe intermittent?

[pleymor]

Looks like it's back! I've got this error for an hour (west europe).

Europe is supposed to be healthy yet according to https://status.dev.azure.com/.

Logs:

##[error]This is a scheduled Ubuntu 16.04 brownout. Ubuntu 16.04 LTS environment will be removed on September 20, 2021. For more details, see https://github.com/actions/virtual-environments/issues/3287
,##[error]The remote provider was unable to process the request.
Pool: Azure Pipelines
Image: ubuntu-16.04
Queued: Just now [manage parallel jobs]
Started: Just now
Duration: 59s

Waiting on the agent to receive and acknowledge the request...
Job preparation parameters
ContinueOnError: False
TimeoutInMinutes: 60
CancelTimeoutInMinutes: 5
Expand:
  MaxConcurrency: 0
  ########## System Pipeline Decorator(s) ##########

  Begin evaluating template 'system-pre-steps.yml'
Evaluating: eq('true', variables['system.debugContext'])
Expanded: eq('true', Null)
Result: False
Evaluating: resources['repositories']['self']
Expanded: Null
Result: False
Finished evaluating template 'system-pre-steps.yml'
********************************************************************************
Template and static variable resolution complete. Final runtime YAML document:
steps: []

[maxim-lobanov]

@pleymor , pay attention on error message:

[error]This is a scheduled Ubuntu 16.04 brownout. Ubuntu 16.04 LTS environment will be removed on September 20, 2021. For more details, see https://github.com/actions/virtual-environments/issues/3287

[pleymor]

Just saw you link, thank you !!

[AlexzSouz]

It appears this issue is back! I'm getting the error below when running my pipeline using agent pool "Azure Pipelines". No changes happened in my azure-pipelines.yml file and just running a simple echo job fails as well.

Error message:

##[warning]An image label with the label Ubuntu16 does not exist.
,##[error]The remote provider was unable to process the request.
Pool: Azure Pipelines
Image: Ubuntu16

Job preparation parameters
Variables:
  FrontendAuthenticationUri: $[ variablegroups.tmo-k8s-variables.FrontendAuthenticationUri ]
  RepositoryUri: $[ variablegroups.tmo-k8s-variables.RepositoryUri ]
  RepositoryUriList: $[ variablegroups.tmo-k8s-variables.RepositoryUriList ]
  TAG: $[ variablegroups.tmo-k8s-variables.TAG ]
  TemplatesUrl: $[ variablegroups.tmo-k8s-variables.TemplatesUrl ]
  VERSION: $[ variablegroups.tmo-k8s-variables.VERSION ]
ContinueOnError: False
TimeoutInMinutes: 60
CancelTimeoutInMinutes: 5
Expand:
  MaxConcurrency: 0
  ########## System Pipeline Decorator(s) ##########

  Begin evaluating template 'system-pre-steps.yml'
Evaluating: eq('true', variables['system.debugContext'])
Expanded: eq('true', Null)
Result: False
Evaluating: resources['repositories']['self']
Expanded: Object
Result: True
Evaluating: not(containsValue(job['steps']['*']['task']['id'], '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Expanded: not(containsValue(Object, '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Result: True
Evaluating: resources['repositories']['self']['checkoutOptions']
Result: Object
Finished evaluating template 'system-pre-steps.yml'
********************************************************************************
Template and static variable resolution complete. Final runtime YAML document:
steps:
- task: 6d15af64-176c-496d-b583-fd2ae21d4df4@1
  inputs:
    repository: self

  MaxConcurrency: 0

YAML Sample:

name: "Backend API $(Build.BuildId)"

variables:
  - group: tmo-k8s-variables

trigger:
  branches:
    include:
      - release/*
      - hotfix/*
      - feature/*
      - main

stages:
  - stage: testing_stage
    displayName: "🧪 Testing"
    jobs:
      - job: testit
        displayName: "Testing"
        steps:
          - bash: |
              echo "Unit tests should run here."
            displayName: "Testing"

Does anyone have an idea? Both Azure status page related to Azure DevOps shows it as healthy.

• https://status.dev.azure.com/

• https://status.azure.com/en-us/status/

[miketimofeev]

@alexz76 Hi! It's related to the Ubuntu16 retirement. Take a look at this issue, please — it contains possible solutions. https://github.com/actions/virtual-environments/issues/4302

[crhistianramirez]

I saw this crop up yesterday and then again today. It seems to occur intermittently.

[gene-keys]

UK West 10-19 and 10-20-21 same issue, ticket is open trying to prove there is an issue on their side (as usual).

[leonardBang]

same issue in South Asia https://dev.azure.com/ververica/flink-cdc-connectors/_build/results?buildId=532&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9

[miketimofeev]

@gene-keys @leonardBang this is related to the Ubuntu-16.04 retirement. Take a look at this thread please https://github.com/actions/virtual-environments/issues/3287 The solutions are at the end of it.

[leonardBang]

Thanks for your @miketimofeev response, I've fixed this issue, and the docs[1] should update too https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml

[gene-keys]

@miketimofeev there is no working solution on that post, already using "vmImage: 'ubuntu-latest'", changing to Azure Pipelines in my region results in 503 service not available. What is likely happening is the issue is regional and intermittent.

[gene-keys]

@miketimofeev the solution with "triggers" that an Azure rep probably suggested in not correct. The drop down simply calls for vmImage: 'ubuntu-latest as its set in yaml and Azure backend returns the wrong version 16, setting it to azure pipelines does not solve the issue. The problem is not closed and is regional as confirmed by Azure support representative, see below, so expect this to pop up again until its patched on Azure side.

"I would like to inform you that we're actively working on a solution for this scenario to update the default agent pool set for a specific pipeline. As you mentioned, this is indeed related to one of our recent changes to the Azure Pipelines agent pool, which was the removal of the ubuntu-16.04 agents from it. This has caused that the pipelines that had the default pool set to it, are facing this issue. Thanks in advance for your patience and comprehension, please keep me updated on the results of it and if the main update has spread to your region."

[Sushma-tiwari]

Mine is Central US check to snap : https://prnt.sc/1y1xjvm

But But still getting this error from past 7 days :

[warning]An image label with the label ubuntu-16.04 does not exist.

,##[error]The remote provider was unable to process the request.

Check error snap : https://prnt.sc/1y1xped

Thanks,

[crhistianramirez]

The ubuntu retirement may be the cause for some users but I am running windows 2019 vm and still seeing it crop up.

[NeerajaGandla]

Same issue here. I don't use Ubuntu. I use Windows 10. Did anyone find a resolution?

[crhistianramirez]

@NeerajaGandla for me it was because I was using azure-pipelines.yml file and while I had defined pool.vmImage for one of my jobs, I had left it blank for two others and for some reason it was pulling in the old failed ubuntu version when not explicitly defined. You need to explicitly set pool.vmImage if you are not currently

Link to the commit that fixed my issue

[BrainSlugs83]

@miketimofeev -- it seems that his issue has regressed today in Central US region.

There is no outage reported on https://status.azure.com/en-us/status/ that I can see. --

1. Can you please update the Azure outage page, or re-open this issue?
2. The error I recieve is the exact same error message and it is very vague. -- It seems the real issue is that the error is not actionable and it will always regress back to this bug because it's the same error message as this one. -- Can you please update the error messages to be less vague so that we can do something actionable? (or where we can open a new bug without it being a duplicate of this one?)

##[error]An error occurred while provisioning resources (Error Type: Failure).
,##[error]The remote provider was unable to process the request.
Pool: [Azure Pipelines](https://dev.azure.com/REDACTED/REDACTED/_settings/agentqueues?poolId=&queueId=27)
Image: windows-latest
Started: Today at 10:11 AM
Duration: 6m 9s

[schafric]

@BrainSlugs83 , @miketimofeev we have the same issue. Status page acknowledges nothing as you wrote @BrainSlugs83 🤷🏻‍♂️

[coolhome]

I can confirm it's been having pretty sporadically in the last few hours for Microsoft Hosted Agents. I wish the pipelines would re-try when this type of error occurs.

[softwareliberationarmy]

Same here. Seeing this issue in the Central US region since yesterday.

[mscammacca-andmore]

This is affecting our org as well, EastUs2 since yesterday.

[stevebumbaugh]

Seeing this in Central US region as well, intermittently today

[paillave]

Since today (the 13th february), in Europe, I face this issue in 75% of my builds. It seems to happen with heavy CPU build (TypeScript for example). It seems that for some reason, the process is lost, therefore, at some point the build process realise after 30 minutes that there is a "Disconnection"

[ahilmathew]

Same issue happening now

[ksooner]

getting same error in Australia region ##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.

[tknaller]

same here west europe

[Lkraehen]

+1 Australia region

[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.

[landre14]

Same here in the US using 'windows-latest' hosted agent

,##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.
,##[error]Provisioning request delayed or failed to send 5 time(s). This is over the limit of 3 time(s).

[michaelcohenunsw]

+1 Australia region

[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.

[shaandre96]

+1 Australia region

[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.

[kchilka-msft]

+1 same here, in US region 🥲

[nanonandi]

Same issue weu :(

[ko16a46]

Is there an official way to report this pipeline host downtime issue to Azure Devops? We're getting the same issue here in East US 2 region but there's no reported downtime in https://status.dev.azure.com/

##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.
,##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.
,##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.
,##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.
,##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider.
,##[error]Provisioning request delayed or failed to send 5 time(s). This is over the limit of 3 time(s).

[tknaller]

my google fu tells me we should be telling this to https://azure.microsoft.com/en-us/support/devops/ - but there's only an error aswell.

[jmeraq]

Mismo problema por aqui !! Heeeeeeeeeeeeeeelp !!

[landre14]

I created an issue with Microsoft here: https://developercommunity.visualstudio.com/t/Azure-Pipeline-hosted-agents-are-down-w/10298058?q=azure+pipeline+hosted+agent

[germanmartin]

We are expecting the same issues! I opened an internal ticket to microsoft with A severity

[Gustavo-Marques19]

same here

##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider. ,##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider. ,##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider. ,##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider. ,##[warning]There was a failure in sending the provision message: An unexpected failure occurred while attempting to contact the remote provider. ,##[error]Provisioning request delayed or failed to send 5 time(s). This is over the limit of 3 time(s). Pool: [Azure Pipelines](https://dev.azure.com/DevOpsVBB/d13ea5be-8198-4358-9a51-c13c09294942/_settings/agentqueues?poolId=&queueId=38) Image: windows-2022 Started: Today at 20:37 Duration: 45m 37s

[hanguyentgg]

+1 Australia

[GFoley83]

Just goes to show, as ever, the Microsoft status page is absolutely USELESS and only updated after everyone knows there's an issue. Infuriating. Issue has nothing to do with Github actions.