[BUG]: API Response missing hashValue for version 3.225.0

[CN120]

What happened?

I have a script that gets the download info for the latest agent version using the api endpoint https://dev.azure.com/{orginization}/_apis/distributedtask/packages/agent and would use the accompanying hashValue to verify the checksum of the download. For whatever reason, the 3.225.0 version of the agent does not have a hashValue provided with it in the api response.

Versions

Azure DevOps Agent version 3.225.0

Environment type (Please select at least one enviroment where you face this issue)

• [ ] Self-Hosted
• [ ] Microsoft Hosted
• [ ] VMSS Pool
• [ ] Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

No response

Relevant log output

response from: https://dev.azure.com/{orginization}/_apis/distributedtask/packages/agent?platform=win-x64&$top=5

{
    "count": 5,
    "value": [
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-08-02T07:27:14.71Z",
            "version": {
                "major": 3,
                "minor": 225,
                "patch": 0
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.225.0/vsts-agent-win-x64-3.225.0.zip",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.225.0.zip"
        },
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-07-25T08:47:09.973Z",
            "version": {
                "major": 3,
                "minor": 224,
                "patch": 1
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.224.1/vsts-agent-win-x64-3.224.1.zip",
            "hashValue": "7f056256946e090b7b8bc125388c8f484e0fe433c5cdc9d01b9995b9ddeac15e",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.224.1.zip"
        },
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-06-06T14:30:53.72Z",
            "version": {
                "major": 3,
                "minor": 220,
                "patch": 5
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.220.5/vsts-agent-win-x64-3.220.5.zip",
            "hashValue": "f8e491f950769ab6664d6e4a346ed78a859aa2975f6d9dd083cbdc4d179bc0c5",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.220.5.zip"
        },
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-05-17T15:08:13.99Z",
            "version": {
                "major": 3,
                "minor": 220,
                "patch": 2
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.220.2/vsts-agent-win-x64-3.220.2.zip",
            "hashValue": "62ac2cc0755a49c59f8fd51c6b6530a9c338252c978f68eb78bad33542e141b3",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.220.2.zip"
        },
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-05-09T08:54:07.95Z",
            "version": {
                "major": 3,
                "minor": 220,
                "patch": 1
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.220.1/vsts-agent-win-x64-3.220.1.zip",
            "hashValue": "fa5aa16bb8623a21c452ba4655a9d6cd632dd55e2105df9419e472a223c2ff8e",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.220.1.zip"
        }
    ]
}

[ivanduplenskikh]

I can't reproduce it. JSON below is the response from the URL:

https://dev.azure.com/{your_organization}/_apis/distributedtask/packages/agent?platform=win-x64&$top=1

{
"count": 1,
"value": [
{
"type": "agent",
"platform": "win-x64",
"createdOn": "2023-07-25T13:12:51.4Z",
"version": {
"major": 3,
"minor": 225,
"patch": 0
},
"downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.225.0/vsts-agent-win-x64-3.225.0.zip",
"hashValue": "f4b643a1b2534198198e431f20631c9be9dcc1550b21b0a0bd2ccc781722670c",
"infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
"filename": "vsts-agent-win-x64-3.225.0.zip"
}
]
}

Besides agent v2.225.0 doesn't have the latest tag and the rollout is in process.

[CN120]

@ivanduplenskikh Is there some way to know which agent does have the latest tag from the API? Because the $top=1 URL is the one we use for determining which one to install on machines (which is the same version that is downloaded when doing so in the Azure DevOps UI for our org). Using the $top=1 call you used, I am still getting no hashValue for our org. I am noticing that the createdOn date is different between our two calls. Did the version created on 8-02-2023 not get a hashValue generated for it?

{
    "count": 1,
    "value": [
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-08-02T07:27:14.71Z",
            "version": {
                "major": 3,
                "minor": 225,
                "patch": 0
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.225.0/vsts-agent-win-x64-3.225.0.zip",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.225.0.zip"
        }
    ]
}

[ivanduplenskikh]

@CN120, this issue is related to Azure DevOps API. This repo is mostly for the agent and I would suggest you open a ticket on https://developercommunity.visualstudio.com/AzureDevOps to get the right eyes on it. I'm closing this one as an external problem - feel free to ask any other questions or let us know if it's still relevant for you.

[adrian-skybaker]

@ivanduplenskikh we've lost all our agents for the same reason, the auto update fails due to 3.225.0 failing checksum validation

[2023-08-14 04:43:55Z INFO Terminal] WRITE LINE: Downloading 3.225.0 agent
[2023-08-14 04:43:55Z INFO HostContext] Well known directory 'Bin': '/home/ec2-user/vsts-agent/bin'
[2023-08-14 04:43:55Z INFO HostContext] Well known directory 'Root': '/home/ec2-user/vsts-agent'
[2023-08-14 04:43:55Z INFO HostContext] Well known directory 'Work': '/home/ec2-user/vsts-agent/_work'
[2023-08-14 04:43:55Z INFO SelfUpdater] Attempt 1: save latest agent into /home/ec2-user/vsts-agent/_work/_update/agent1.tar.gz.
[2023-08-14 04:43:55Z INFO SelfUpdater] Download agent: begin download
[2023-08-14 04:43:56Z INFO SelfUpdater] Download agent: finished download
[2023-08-14 04:43:56Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2023-08-14 04:43:56Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2023-08-14 04:43:56Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2023-08-14 04:43:57Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2023-08-14 04:43:57Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed

[ivanduplenskikh]

@CN120, @adrian-skybaker, could you please provide your organization names?

[adrian-skybaker]

Sky-Digital

[FreshPrinceMayo]

We face the same issue can't update agents

Agent Logs

[2023-08-15 08:53:48Z INFO SelfUpdater] An update is available. [2023-08-15 08:53:48Z INFO Terminal] WRITE LINE: Agent update in progress, do not shutdown agent. [2023-08-15 08:53:48Z INFO Terminal] WRITE LINE: Downloading 3.225.0 agent [2023-08-15 08:53:48Z INFO HostContext] Well known directory 'Bin': 'A:\xxxxx-DevOps\bin' [2023-08-15 08:53:48Z INFO HostContext] Well known directory 'Root': 'A:\xxxxx-DevOps' [2023-08-15 08:53:48Z INFO HostContext] Well known directory 'Work': 'A:\xxxxx-DevOps\_work' [2023-08-15 08:53:48Z INFO SelfUpdater] Attempt 1: save latest agent into A:\xxxxx-DevOps\_work\_update\agent1.zip. [2023-08-15 08:53:48Z INFO SelfUpdater] Download agent: begin download [2023-08-15 08:53:57Z INFO SelfUpdater] Download agent: finished download [2023-08-15 08:53:57Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed [2023-08-15 08:53:57Z INFO SelfUpdater] Attempt 2: save latest agent into A:\xxxxx-DevOps\_work\_update\agent1.zip. [2023-08-15 08:53:57Z INFO SelfUpdater] Download agent: begin download [2023-08-15 08:54:05Z INFO SelfUpdater] Download agent: finished download [2023-08-15 08:54:05Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed [2023-08-15 08:54:05Z INFO SelfUpdater] Attempt 3: save latest agent into A:\xxxxx-DevOps\_work\_update\agent1.zip. [2023-08-15 08:54:05Z INFO SelfUpdater] Download agent: begin download [2023-08-15 08:54:24Z INFO SelfUpdater] Download agent: finished download [2023-08-15 08:54:24Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed [2023-08-15 08:54:24Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished. [2023-08-15 08:54:24Z ERR Terminal] WRITE ERROR: An error occurred: Agent package checksum validation failed. There are possible reasons why this happened: 1) The agent package was compromised. 2) The agent package was not fully downloaded or was corrupted during the download process. You can skip checksum validation for the agent package by setting the environment variable DISABLE_HASH_VALIDATION=true [2023-08-15 08:54:24Z ERR AgentProcess] System.Threading.Tasks.TaskCanceledException: Agent package checksum validation failed. There are possible reasons why this happened: 1) The agent package was compromised. 2) The agent package was not fully downloaded or was corrupted during the download process. You can skip checksum validation for the agent package by setting the environment variable DISABLE_HASH_VALIDATION=true at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.DownloadLatestAgent(CancellationToken token) at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.SelfUpdate(AgentRefreshMessage updateMessage, IJobDispatcher jobDispatcher, Boolean restartInteractiveAgent, CancellationToken token) at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) at Microsoft.VisualStudio.Services.Agent.Listener.Agent.ExecuteCommand(CommandSettings command) at Microsoft.VisualStudio.Services.Agent.Listener.Program.MainAsync(IHostContext context, String[] args)

Link to Issue on developer community: Update of self-hosted agents v3.224.1 to v3.225.0 is not working from Azure DevOps site

[adrian-skybaker]

My workaround was to ssh into the agents, edit ~/vsts-agent/.env, add DISABLE_HASH_VALIDATION=true, and restart.

Though if you regularly tear down/recreate your agents (eg autoscaling, spot instances etc), then you'd have to keep doing this, or update your image.

[njkoopman]

We also have the same problem: { "count": 1, "value": [{ "type": "agent", "platform": "win-x64", "createdOn": "2023-08-15T07:30:20.81Z", "version": { "major": 3, "minor": 225, "patch": 0 }, "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.225.0/vsts-agent-win-x64-3.225.0.zip", "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199", "filename": "vsts-agent-win-x64-3.225.0.zip" } ] }

[CCargan]

We also have the same problem. I wonder why that we get a version for download that are in a "pre-lease". I think that "Latest" should be the one for download.

[ivanduplenskikh]

@CN120, @adrian-skybaker, @FreshPrinceMayo, @njkoopman, @CCargan, we have completed the agent release, now v3.225.0 is the latest. Could you please confirm the availability of the hashValue property?

[FreshPrinceMayo]

Working for us now

[njkoopman]

It works for us as well, all agents have been updated without problems. Thank for the fix!