[BUG]: Azure IoT Edge v2 - Failed to fetch container registry authentication token

magneticore commented 1 week ago

New issue checklist

[X] I searched for existing GitHub issues
[X] I read pipeline troubleshooting guide
[X] I checked how to collect logs

Task name

Azure IoT Edge

Task version

2

Issue Description

Hello!

The Azure IoT Edge - Push module image task is failing.

For context: Our last successful push was on June 14 as part of the validation steps in our CI/CD pipeline. These steps execute on PR creation as a requirement for the PR submission. Unfortunately for debugging means that all the output of the run is deleted since is not yet part of one of our designated branches. "This run is scheduled to be cleaned up the next time the cleanup job executes"

Since Monday we get an authentication token error.

Also any new builds or pipeline runs even using PRs when the pipeline was green now return the following message:

2024-06-18T20:17:02.2873393Z ##[section]Starting: Azure IoT Edge - Push module image 2024-06-18T20:17:02.2879784Z ============================================================================== 2024-06-18T20:17:02.2879934Z Task : Azure IoT Edge 2024-06-18T20:17:02.2880004Z Description : Build and deploy an Azure IoT Edge image 2024-06-18T20:17:02.2880115Z Version : 2.241.1 2024-06-18T20:17:02.2880180Z Author : Microsoft Corporation 2024-06-18T20:17:02.2880435Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge 2024-06-18T20:17:02.2880796Z ============================================================================== 2024-06-18T20:17:02.8225767Z Pushing module images... 2024-06-18T20:17:02.8293944Z ##[error]Failed to fetch container registry authentication token, please check you container registry setting in build task. The username for container registry is 2024-06-18T20:17:03.0065404Z ##[section]Finishing: Azure IoT Edge - Push module image

Checking this repo for the error I noticed the missing value on the string for username.

"InvalidContainerRegistry": "Failed to fetch container registry authentication token, please check you container registry setting in build task. The username for container registry is %s"

Extra debugging steps:

I added a bash task before this one as a way to debug and the ACR information shows up including the username.

As a second debugging method I added a Docker login task for the same ACR, the result also succeeds.

The original run was on Ubuntu-latest I also tested with Ubuntu-20.04, the error is present in both.

Environment type (Please select at least one enviroment where you face this issue)

[ ] Self-Hosted
[X] Microsoft Hosted
[ ] VMSS Pool
[ ] Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu-latest and Ubuntu-20.04

Relevant log output

ERROR example:

2024-06-18T20:17:02.2873393Z ##[section]Starting: Azure IoT Edge - Push module image
2024-06-18T20:17:02.2879784Z ==============================================================================

2024-06-18T20:17:02.2879934Z Task         : Azure IoT Edge
2024-06-18T20:17:02.2880004Z Description  : Build and deploy an Azure IoT Edge image
2024-06-18T20:17:02.2880115Z Version      : 2.241.1
2024-06-18T20:17:02.2880180Z Author       : Microsoft Corporation
2024-06-18T20:17:02.2880435Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
2024-06-18T20:17:02.2880796Z ==============================================================================
2024-06-18T20:17:02.8225767Z Pushing module images...
2024-06-18T20:17:02.8293944Z ##[error]Failed to fetch container registry authentication token, please check you container registry setting in build task. The username for container registry is 
2024-06-18T20:17:03.0065404Z ##[section]Finishing: Azure IoT Edge - Push module image

Compared to our last successful recorded run:
______________________________________________

Starting: Azure IoT Edge - Push module image
==============================================================================
Task         : Azure IoT Edge
Description  : Build and deploy an Azure IoT Edge image
Version      : 2.238.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
==============================================================================
/home/vsts/work/1/s/EdgeSolution /home/vsts/work/1/s
Pushing module images...
iotedgedev already installed with version: version 3.3.7

/opt/hostedtoolcache/Python/3.8.18/x64/bin/iotedgedev "push" "--no-build" "--file" "/home/vsts/work/1/s/EdgeSolution/deployment.template.json" "--platform" "arm32v7"

=======================================
======== ENVIRONMENT VARIABLES ========
=======================================

.env file not found on disk. Without a file on disk, you must specify all Environment Variables at the system level. (/home/vsts/work/1/s/EdgeSolution/.env)
DeprecationWarning: The command 'push' is deprecated.

=================================================
======== INITIALIZING CONTAINER REGISTRY ========
=================================================

REGISTRY: MyACR_NAME.azurecr.io

PUSHING DOCKER IMAGE: MyACR_NAME.azurecr.io/MyMODULE:20240603.2-0.1.2-arm32v7
...
408743bc2c8a Pushing  1.536kB
4cc798763347 Pushing [=>                                                 ]     512B/16.42kB

Full task logs with system.debug enabled

 2024-06-18T16:27:14.8429978Z ##[debug]Evaluating condition for step: 'Azure IoT Edge - Push module image'
2024-06-18T16:27:14.8430862Z ##[debug]Evaluating: SucceededNode()
2024-06-18T16:27:14.8431101Z ##[debug]Evaluating SucceededNode:
2024-06-18T16:27:14.8431530Z ##[debug]=> True
2024-06-18T16:27:14.8431771Z ##[debug]Result: True
2024-06-18T16:27:14.8432038Z ##[section]Starting: Azure IoT Edge - Push module image
2024-06-18T16:27:14.8437877Z ==============================================================================
2024-06-18T16:27:14.8438016Z Task         : Azure IoT Edge
2024-06-18T16:27:14.8438089Z Description  : Build and deploy an Azure IoT Edge image
2024-06-18T16:27:14.8438194Z Version      : 2.241.1
2024-06-18T16:27:14.8438259Z Author       : Microsoft Corporation
2024-06-18T16:27:14.8438335Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
2024-06-18T16:27:14.8438460Z ==============================================================================
2024-06-18T16:27:14.8854288Z ##[debug]Resource Utilization warnings is disabled, switch "DISABLE_RESOURCE_UTILIZATION_WARNINGS" variable to "true" if you want to enable it
2024-06-18T16:27:14.9219733Z ##[debug]Using node path: /home/vsts/agents/3.240.1/externals/node20_1/bin/node
2024-06-18T16:27:15.4385777Z ##[debug]agent.TempDirectory=/home/vsts/work/_temp
2024-06-18T16:27:15.4386168Z ##[debug]loading inputs and endpoints
2024-06-18T16:27:15.4386502Z ##[debug]loading INPUT_ACTION
2024-06-18T16:27:15.4386823Z ##[debug]loading INPUT_DEPLOYMENTFILEPATH
2024-06-18T16:27:15.4387153Z ##[debug]loading INPUT_DEPLOYMENTID
2024-06-18T16:27:15.4387491Z ##[debug]loading INPUT_PRIORITY
2024-06-18T16:27:15.4387815Z ##[debug]loading INPUT_CONTAINERREGISTRYTYPE
2024-06-18T16:27:15.4388158Z ##[debug]loading INPUT_AZURESUBSCRIPTIONENDPOINT
2024-06-18T16:27:15.4388496Z ##[debug]loading INPUT_AZURECONTAINERREGISTRY
2024-06-18T16:27:15.4388850Z ##[debug]loading INPUT_TEMPLATEFILEPATH
2024-06-18T16:27:15.4389178Z ##[debug]loading INPUT_DEFAULTPLATFORM
2024-06-18T16:27:15.4389519Z ##[debug]loading INPUT_FILLREGISTRYCREDENTIAL
2024-06-18T16:27:15.4389876Z ##[debug]loading INPUT_DEPLOYMENTMANIFESTOUTPUTPATH
2024-06-18T16:27:15.4390234Z ##[debug]loading INPUT_VALIDATEGENERATEDDEPLOYMENTMANIFEST
2024-06-18T16:27:15.4390604Z ##[debug]loading ENDPOINT_AUTH_8c07e9fb-8c00-4231-9abb-9fe9b54c9060
2024-06-18T16:27:15.4391006Z ##[debug]loading ENDPOINT_AUTH_SCHEME_8c07e9fb-8c00-4231-9abb-9fe9b54c9060
2024-06-18T16:27:15.4391403Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_8c07e9fb-8c00-4231-9abb-9fe9b54c9060_TENANTID
2024-06-18T16:27:15.4391835Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_8c07e9fb-8c00-4231-9abb-9fe9b54c9060_SERVICEPRINCIPALID
2024-06-18T16:27:15.4392265Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_8c07e9fb-8c00-4231-9abb-9fe9b54c9060_SERVICEPRINCIPALKEY
2024-06-18T16:27:15.4392654Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2024-06-18T16:27:15.4393287Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2024-06-18T16:27:15.4393664Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2024-06-18T16:27:15.4394027Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
2024-06-18T16:27:15.4394383Z ##[debug]loading SECRET_CONTAINER_REGISTRY_PASSWORD
2024-06-18T16:27:15.4394704Z ##[debug]loaded 22
2024-06-18T16:27:15.4395008Z ##[debug]Agent.ProxyUrl=undefined
2024-06-18T16:27:15.4395324Z ##[debug]Agent.CAInfo=undefined
2024-06-18T16:27:15.4395661Z ##[debug]Agent.ClientCert=undefined
2024-06-18T16:27:15.4395989Z ##[debug]Agent.SkipCertValidation=undefined
2024-06-18T16:27:15.4396316Z ##[debug]agent.proxyurl=undefined
2024-06-18T16:27:15.4396662Z ##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
2024-06-18T16:27:15.4397044Z ##[debug]AZURE_HTTP_USER_AGENT=VSTS_0defe19c-c87e-4be5-9176-866f6e6a1884_build_148_0
2024-06-18T16:27:15.4397519Z ##[debug]check path : /home/vsts/work/_tasks/AzureIoTEdge_80f3f6a0-82a6-4a22-ba7a-e5b8c541b9b8/2.241.1/node_modules/azure-pipelines-tasks-azure-arm-rest/module.json
2024-06-18T16:27:15.4398083Z ##[debug]adding resource file: /home/vsts/work/_tasks/AzureIoTEdge_80f3f6a0-82a6-4a22-ba7a-e5b8c541b9b8/2.241.1/node_modules/azure-pipelines-tasks-azure-arm-rest/module.json
2024-06-18T16:27:15.4398688Z ##[debug]system.culture=en-US
2024-06-18T16:27:15.4399078Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp
2024-06-18T16:27:15.4399535Z ##[debug]Resource file has already set to: /home/vsts/work/_tasks/AzureIoTEdge_80f3f6a0-82a6-4a22-ba7a-e5b8c541b9b8/2.241.1/node_modules/azure-pipelines-tasks-azure-arm-rest/module.json
2024-06-18T16:27:15.4400104Z ##[debug]check path : /home/vsts/work/_tasks/AzureIoTEdge_80f3f6a0-82a6-4a22-ba7a-e5b8c541b9b8/2.241.1/node_modules/azure-pipelines-tasks-artifacts-common/module.json
2024-06-18T16:27:15.4400650Z ##[debug]adding resource file: /home/vsts/work/_tasks/AzureIoTEdge_80f3f6a0-82a6-4a22-ba7a-e5b8c541b9b8/2.241.1/node_modules/azure-pipelines-tasks-artifacts-common/module.json
2024-06-18T16:27:15.4401084Z ##[debug]system.culture=en-US
2024-06-18T16:27:15.4401552Z ##[debug]Resource file has already set to: /home/vsts/work/_tasks/AzureIoTEdge_80f3f6a0-82a6-4a22-ba7a-e5b8c541b9b8/2.241.1/node_modules/azure-pipelines-tasks-artifacts-common/module.json
2024-06-18T16:27:15.4402061Z ##[debug]check path : /home/vsts/work/_tasks/AzureIoTEdge_80f3f6a0-82a6-4a22-ba7a-e5b8c541b9b8/2.241.1/task.json
2024-06-18T16:27:15.4402528Z ##[debug]adding resource file: /home/vsts/work/_tasks/AzureIoTEdge_80f3f6a0-82a6-4a22-ba7a-e5b8c541b9b8/2.241.1/task.json
2024-06-18T16:27:15.4402912Z ##[debug]system.culture=en-US
2024-06-18T16:27:15.4403218Z ##[debug]which 'lsb_release'
2024-06-18T16:27:15.4403544Z ##[debug]found: '/usr/bin/lsb_release'
2024-06-18T16:27:15.4403862Z ##[debug]/usr/bin/lsb_release arg: -a
2024-06-18T16:27:15.4404177Z ##[debug]/usr/bin/lsb_release arg: -a
2024-06-18T16:27:15.4404521Z ##[debug]exec tool: /usr/bin/lsb_release
2024-06-18T16:27:15.4404844Z ##[debug]exec tool: /usr/bin/lsb_release
2024-06-18T16:27:15.4405152Z ##[debug]arguments:
2024-06-18T16:27:15.4405441Z ##[debug]arguments:
2024-06-18T16:27:15.4405741Z ##[debug]   -a
2024-06-18T16:27:15.4406019Z ##[debug]   -a
2024-06-18T16:27:15.4406395Z ##[debug]OS is Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:    22.04
Codename:   jammy

2024-06-18T16:27:15.4406779Z ##[debug]action=Push module images
2024-06-18T16:27:15.4407129Z ##[debug]system.teamProjectId=dcbd204b-0475-443d-bc8f-bb49b73a6bbe
2024-06-18T16:27:15.4407470Z ##[debug]build.buildId=112661
2024-06-18T16:27:15.4407793Z ##[debug]System.ServerType=Hosted
2024-06-18T16:27:15.4408115Z ##[debug]DISABLE_TELEMETRY=undefined
2024-06-18T16:27:15.4408433Z ##[debug]System.ServerType=Hosted
2024-06-18T16:27:15.4408643Z Pushing module images...
2024-06-18T16:27:15.4408932Z ##[debug]containerregistrytype=Azure Container Registry
2024-06-18T16:27:15.4409299Z ##[debug]containerregistrytype=Azure Container Registry
2024-06-18T16:27:15.4409656Z ##[debug]azureSubscriptionEndpointForSecrets=undefined
2024-06-18T16:27:15.4410125Z ##[debug]azureContainerRegistry={"loginServer":"MyACR.azurecr.io"}
2024-06-18T16:27:15.4410486Z ##[debug]task result: Failed
2024-06-18T16:27:15.4440595Z ##[error]Failed to fetch container registry authentication token, please check you container registry setting in build task. The username for container registry is 
2024-06-18T16:27:15.4447057Z ##[debug]Processed: ##vso[task.issue type=error;]Failed to fetch container registry authentication token, please check you container registry setting in build task. The username for container registry is 
2024-06-18T16:27:15.4448149Z ##[debug]Processed: ##vso[task.complete result=Failed;]Failed to fetch container registry authentication token, please check you container registry setting in build task. The username for container registry is 
2024-06-18T16:27:15.4448620Z ##[debug]Agent.Version=3.240.1
2024-06-18T16:27:15.4448995Z ##[debug]SYSTEM_TASKINSTANCEID=f6bcc2a0-dcd5-5f83-7714-1a03c50147ea
2024-06-18T16:27:15.4449391Z ##[debug]SYSTEM_JOBID=3474c78a-f2e4-54e8-5e18-4c6ce8fab9c4
2024-06-18T16:27:15.4449758Z ##[debug]SYSTEM_PLANID=35727e59-be92-4182-a362-cd1747af957b
2024-06-18T16:27:15.4450409Z ##[debug]SYSTEM_COLLECTIONID=0defe19c-c87e-4be5-9176-866f6e6a1884
2024-06-18T16:27:15.4450746Z ##[debug]AGENT_ID=32
2024-06-18T16:27:15.4451060Z ##[debug]AGENT_MACHINENAME=fv-az705-807
2024-06-18T16:27:15.4451383Z ##[debug]AGENT_NAME=Hosted Agent
2024-06-18T16:27:15.4451716Z ##[debug]AGENT_JOBSTATUS=Succeeded
2024-06-18T16:27:15.4452029Z ##[debug]AGENT_OS=Linux
2024-06-18T16:27:15.4452334Z ##[debug]AGENT_VERSION=3.240.1
2024-06-18T16:27:15.4452654Z ##[debug]BUILD_BUILDID=112661
2024-06-18T16:27:15.4452972Z ##[debug]BUILD_BUILDNUMBER=20240618.1
2024-06-18T16:27:15.4453308Z ##[debug]BUILD_BUILDURI=vstfs:///Build/Build/112661
2024-06-18T16:27:15.4453654Z ##[debug]BUILD_CONTAINERID=64413218
2024-06-18T16:27:15.4453991Z ##[debug]BUILD_DEFINITIONNAME=CI - MyModule
2024-06-18T16:27:15.4454324Z ##[debug]BUILD_DEFINITIONVERSION=5
2024-06-18T16:27:15.4454649Z ##[debug]BUILD_REASON=Manual
2024-06-18T16:27:15.4454964Z ##[debug]BUILD_REPOSITORY_CLEAN=False
2024-06-18T16:27:15.4455302Z ##[debug]BUILD_REPOSITORY_GIT_SUBMODULECHECKOUT=False
2024-06-18T16:27:15.4455660Z ##[debug]BUILD_REPOSITORY_NAME=MyModule
2024-06-18T16:27:15.4455995Z ##[debug]BUILD_REPOSITORY_PROVIDER=TfsGit
2024-06-18T16:27:15.4456352Z ##[debug]BUILD_SOURCEVERSION=10a8f04717be3bc43eff5a9e8b5697697210d8ed
2024-06-18T16:27:15.4763594Z ##[debug]Processed: ##vso[telemetry.publish area=TaskEndpointId;feature=AzureIoTEdgeV2]{"SYSTEM_TASKINSTANCEID":"f6bcc2a0-dcd5-5f83-7714-1a03c50147ea","SYSTEM_JOBID":"3474c78a-f2e4-54e8-5e18-4c6ce8fab9c4","SYSTEM_PLANID":"35727e59-be92-4182-a362-cd1747af957b","SYSTEM_COLLECTIONID":"0defe19c-c87e-4be5-9176-866f6e6a1884","AGENT_ID":"32","AGENT_MACHINENAME":"fv-az705-807","AGENT_NAME":"Hosted Agent","AGENT_JOBSTATUS":"Succeeded","AGENT_OS":"Linux","AGENT_VERSION":"3.240.1","BUILD_BUILDID":"112661","BUILD_BUILDNUMBER":"20240618.1","BUILD_BUILDURI":"vstfs:///Build/Build/112661","BUILD_CONTAINERID":"64413218","BUILD_DEFINITIONNAME":"CI - gw_hub_client_edge","BUILD_DEFINITIONVERSION":"5","BUILD_REASON":"Manual","BUILD_REPOSITORY_CLEAN":"False","BUILD_REPOSITORY_GIT_SUBMODULECHECKOUT":"False","BUILD_REPOSITORY_NAME":"gw_hub_client_edge","BUILD_REPOSITORY_PROVIDER":"TfsGit","BUILD_SOURCEVERSION":"10a8f04717be3bc43eff5a9e8b5697697210d8ed","hashTeamProjectId":"c3cb820fbdca075b100ef9f7e564e7dedde90805e1b254e705852b3147ffc945","taskType":"Push module images","osType":"Linux","buildId":"112661","isSuccess":false,"taskTime":0.004,"serverType":"Hosted","fixedCliExtInstalled":null,"error":"Failed to fetch container registry authentication token","isACR":true}
2024-06-18T16:27:15.4878375Z ##[section]Finishing: Azure IoT Edge - Push module image

Repro steps

trigger:
- master
- develop

resources:
- repo: self

variables:
  VIRTUAL_ENV: 'venv'
  vmImageName: 'ubuntu-20.04'

pool:
  vmImage: $(vmImageName)
stages:
  - stage: IoT_Edge_Module
    displayName: Edge module build and push
    jobs:
    - job: Build
      displayName: Pull Docker images
      steps:
        - task: AzureIoTEdge@2
          displayName: 'Azure IoT Edge - Build module image'
          inputs:
            action: 'Build module images'
            templateFilePath: 'EdgeSolution/deployment.template.json'
            defaultPlatform: 'arm32v7'
        - bash: |
            echo "ACR address: $(CONTAINER_REGISTRY_ADDRESS)"
            echo "ACR username: $(CONTAINER_REGISTRY_USERNAME)"
        - task: AzureIoTEdge@2
          displayName: 'Azure IoT Edge - Push module image'
          inputs:
            action: 'Push module images'
            containerregistrytype: 'Azure Container Registry'
            azureSubscriptionEndpoint: 'ARM-Service-Connection'
            azureContainerRegistry: '{"loginServer":"MyACR_NAME.azurecr.io"}'
            templateFilePath: 'EdgeSolution/deployment.template.json'
            defaultPlatform: 'arm32v7'
            fillRegistryCredential: 'true'

andrewsuiter commented 1 week ago

I am having the same issue, this just started on 6/18

[error]Failed to fetch container registry authentication token, please check you container registry setting in build task. The username for container registry is

doakcovington-asg commented 1 week ago

FYI looks like this was the breaking change: https://github.com/microsoft/azure-pipelines-tasks/commit/1741517c4b2439116703dc26e8513303ade77d93

Rolling back to version 2.238.1 worked for us

karakal commented 1 week ago

FYI looks like this was the breaking change: 1741517

Rolling back to version 2.238.1 worked for us

How do I do that in Azure DevOps?

JoLanckriet commented 1 week ago

FYI looks like this was the breaking change: 1741517 Rolling back to version 2.238.1 worked for us

How do I do that in Azure DevOps?

Had the same problem and didn't know that either so i tried this: Seems to work for me now!

karakal commented 1 week ago

FYI looks like this was the breaking change: 1741517 Rolling back to version 2.238.1 worked for us

How do I do that in Azure DevOps?

Had the same problem and didn't know that either so i tried this: Seems to work for me now!

I am really too dumb for this. I don't have a azure-pipeline.yml that I can modify? And I can't change the YAML on the DevOps-Page, just view it...

JoLanckriet commented 1 week ago

FYI looks like this was the breaking change: 1741517 Rolling back to version 2.238.1 worked for us

How do I do that in Azure DevOps?

Had the same problem and didn't know that either so i tried this: Seems to work for me now!

I am really too dumb for this. I don't have a azure-pipeline.yml that I can modify? And I can't change the YAML on the DevOps-Page, just view it...

I suppose you have insignificant permissions then? As far as I know you should mostly be able to change the yaml of your pipeline either in the repository or by pressing the 'edit' button in devops. But I should mention I'm also quite new at all of this do I could be wrong.

MartonBoda commented 2 days ago

Reverting the push task to 2.238.1 worked for me. Previously I've tried converting the service connection to use federated credentials, but that had no effect. The workaround only worked after reverting the conversion as well.

microsoft / azure-pipelines-tasks