microsoft / azure-pipelines-agent

Azure Pipelines Agent 🚀
MIT License
1.72k stars 866 forks source link

[BUG]: API Response missing hashValue for version 3.225.0 #4380

Closed CN120 closed 1 year ago

CN120 commented 1 year ago

What happened?

I have a script that gets the download info for the latest agent version using the api endpoint https://dev.azure.com/{orginization}/_apis/distributedtask/packages/agent and would use the accompanying hashValue to verify the checksum of the download. For whatever reason, the 3.225.0 version of the agent does not have a hashValue provided with it in the api response.

Versions

Azure DevOps Agent version 3.225.0

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

No response

Relevant log output

response from: https://dev.azure.com/{orginization}/_apis/distributedtask/packages/agent?platform=win-x64&$top=5

{
    "count": 5,
    "value": [
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-08-02T07:27:14.71Z",
            "version": {
                "major": 3,
                "minor": 225,
                "patch": 0
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.225.0/vsts-agent-win-x64-3.225.0.zip",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.225.0.zip"
        },
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-07-25T08:47:09.973Z",
            "version": {
                "major": 3,
                "minor": 224,
                "patch": 1
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.224.1/vsts-agent-win-x64-3.224.1.zip",
            "hashValue": "7f056256946e090b7b8bc125388c8f484e0fe433c5cdc9d01b9995b9ddeac15e",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.224.1.zip"
        },
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-06-06T14:30:53.72Z",
            "version": {
                "major": 3,
                "minor": 220,
                "patch": 5
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.220.5/vsts-agent-win-x64-3.220.5.zip",
            "hashValue": "f8e491f950769ab6664d6e4a346ed78a859aa2975f6d9dd083cbdc4d179bc0c5",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.220.5.zip"
        },
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-05-17T15:08:13.99Z",
            "version": {
                "major": 3,
                "minor": 220,
                "patch": 2
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.220.2/vsts-agent-win-x64-3.220.2.zip",
            "hashValue": "62ac2cc0755a49c59f8fd51c6b6530a9c338252c978f68eb78bad33542e141b3",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.220.2.zip"
        },
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-05-09T08:54:07.95Z",
            "version": {
                "major": 3,
                "minor": 220,
                "patch": 1
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.220.1/vsts-agent-win-x64-3.220.1.zip",
            "hashValue": "fa5aa16bb8623a21c452ba4655a9d6cd632dd55e2105df9419e472a223c2ff8e",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.220.1.zip"
        }
    ]
}
ivanduplenskikh commented 1 year ago

I can't reproduce it. JSON below is the response from the URL:

https://dev.azure.com/{your_organization}/_apis/distributedtask/packages/agent?platform=win-x64&$top=1

{
"count": 1,
"value": [
{
"type": "agent",
"platform": "win-x64",
"createdOn": "2023-07-25T13:12:51.4Z",
"version": {
"major": 3,
"minor": 225,
"patch": 0
},
"downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.225.0/vsts-agent-win-x64-3.225.0.zip",
"hashValue": "f4b643a1b2534198198e431f20631c9be9dcc1550b21b0a0bd2ccc781722670c",
"infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
"filename": "vsts-agent-win-x64-3.225.0.zip"
}
]
}

Besides agent v2.225.0 doesn't have the latest tag and the rollout is in process.

CN120 commented 1 year ago

@ivanduplenskikh Is there some way to know which agent does have the latest tag from the API? Because the $top=1 URL is the one we use for determining which one to install on machines (which is the same version that is downloaded when doing so in the Azure DevOps UI for our org). Using the $top=1 call you used, I am still getting no hashValue for our org. I am noticing that the createdOn date is different between our two calls. Did the version created on 8-02-2023 not get a hashValue generated for it?

{
    "count": 1,
    "value": [
        {
            "type": "agent",
            "platform": "win-x64",
            "createdOn": "2023-08-02T07:27:14.71Z",
            "version": {
                "major": 3,
                "minor": 225,
                "patch": 0
            },
            "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.225.0/vsts-agent-win-x64-3.225.0.zip",
            "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199",
            "filename": "vsts-agent-win-x64-3.225.0.zip"
        }
    ]
}
ivanduplenskikh commented 1 year ago

@CN120, this issue is related to Azure DevOps API. This repo is mostly for the agent and I would suggest you open a ticket on https://developercommunity.visualstudio.com/AzureDevOps to get the right eyes on it. I'm closing this one as an external problem - feel free to ask any other questions or let us know if it's still relevant for you.

adrian-skybaker commented 1 year ago

@ivanduplenskikh we've lost all our agents for the same reason, the auto update fails due to 3.225.0 failing checksum validation

[2023-08-14 04:43:55Z INFO Terminal] WRITE LINE: Downloading 3.225.0 agent
[2023-08-14 04:43:55Z INFO HostContext] Well known directory 'Bin': '/home/ec2-user/vsts-agent/bin'
[2023-08-14 04:43:55Z INFO HostContext] Well known directory 'Root': '/home/ec2-user/vsts-agent'
[2023-08-14 04:43:55Z INFO HostContext] Well known directory 'Work': '/home/ec2-user/vsts-agent/_work'
[2023-08-14 04:43:55Z INFO SelfUpdater] Attempt 1: save latest agent into /home/ec2-user/vsts-agent/_work/_update/agent1.tar.gz.
[2023-08-14 04:43:55Z INFO SelfUpdater] Download agent: begin download
[2023-08-14 04:43:56Z INFO SelfUpdater] Download agent: finished download
[2023-08-14 04:43:56Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2023-08-14 04:43:56Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2023-08-14 04:43:56Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2023-08-14 04:43:57Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2023-08-14 04:43:57Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed
ivanduplenskikh commented 1 year ago

@CN120, @adrian-skybaker, could you please provide your organization names?

adrian-skybaker commented 1 year ago

Sky-Digital

FreshPrinceMayo commented 1 year ago

We face the same issue can't update agents

Agent Logs

[2023-08-15 08:53:48Z INFO SelfUpdater] An update is available. [2023-08-15 08:53:48Z INFO Terminal] WRITE LINE: Agent update in progress, do not shutdown agent. [2023-08-15 08:53:48Z INFO Terminal] WRITE LINE: Downloading 3.225.0 agent [2023-08-15 08:53:48Z INFO HostContext] Well known directory 'Bin': 'A:\xxxxx-DevOps\bin' [2023-08-15 08:53:48Z INFO HostContext] Well known directory 'Root': 'A:\xxxxx-DevOps' [2023-08-15 08:53:48Z INFO HostContext] Well known directory 'Work': 'A:\xxxxx-DevOps\_work' [2023-08-15 08:53:48Z INFO SelfUpdater] Attempt 1: save latest agent into A:\xxxxx-DevOps\_work\_update\agent1.zip. [2023-08-15 08:53:48Z INFO SelfUpdater] Download agent: begin download [2023-08-15 08:53:57Z INFO SelfUpdater] Download agent: finished download [2023-08-15 08:53:57Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed [2023-08-15 08:53:57Z INFO SelfUpdater] Attempt 2: save latest agent into A:\xxxxx-DevOps\_work\_update\agent1.zip. [2023-08-15 08:53:57Z INFO SelfUpdater] Download agent: begin download [2023-08-15 08:54:05Z INFO SelfUpdater] Download agent: finished download [2023-08-15 08:54:05Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed [2023-08-15 08:54:05Z INFO SelfUpdater] Attempt 3: save latest agent into A:\xxxxx-DevOps\_work\_update\agent1.zip. [2023-08-15 08:54:05Z INFO SelfUpdater] Download agent: begin download [2023-08-15 08:54:24Z INFO SelfUpdater] Download agent: finished download [2023-08-15 08:54:24Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed [2023-08-15 08:54:24Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished. [2023-08-15 08:54:24Z ERR Terminal] WRITE ERROR: An error occurred: Agent package checksum validation failed. There are possible reasons why this happened: 1) The agent package was compromised. 2) The agent package was not fully downloaded or was corrupted during the download process. You can skip checksum validation for the agent package by setting the environment variable DISABLE_HASH_VALIDATION=true [2023-08-15 08:54:24Z ERR AgentProcess] System.Threading.Tasks.TaskCanceledException: Agent package checksum validation failed. There are possible reasons why this happened: 1) The agent package was compromised. 2) The agent package was not fully downloaded or was corrupted during the download process. You can skip checksum validation for the agent package by setting the environment variable DISABLE_HASH_VALIDATION=true at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.DownloadLatestAgent(CancellationToken token) at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.SelfUpdate(AgentRefreshMessage updateMessage, IJobDispatcher jobDispatcher, Boolean restartInteractiveAgent, CancellationToken token) at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) at Microsoft.VisualStudio.Services.Agent.Listener.Agent.ExecuteCommand(CommandSettings command) at Microsoft.VisualStudio.Services.Agent.Listener.Program.MainAsync(IHostContext context, String[] args)

Link to Issue on developer community: Update of self-hosted agents v3.224.1 to v3.225.0 is not working from Azure DevOps site

adrian-skybaker commented 1 year ago

My workaround was to ssh into the agents, edit ~/vsts-agent/.env, add DISABLE_HASH_VALIDATION=true, and restart.

Though if you regularly tear down/recreate your agents (eg autoscaling, spot instances etc), then you'd have to keep doing this, or update your image.

njkoopman commented 1 year ago

We also have the same problem: { "count": 1, "value": [{ "type": "agent", "platform": "win-x64", "createdOn": "2023-08-15T07:30:20.81Z", "version": { "major": 3, "minor": 225, "patch": 0 }, "downloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.225.0/vsts-agent-win-x64-3.225.0.zip", "infoUrl": "https://go.microsoft.com/fwlink/?LinkId=798199", "filename": "vsts-agent-win-x64-3.225.0.zip" } ] }

CCargan commented 1 year ago

We also have the same problem. I wonder why that we get a version for download that are in a "pre-lease". I think that "Latest" should be the one for download.

ivanduplenskikh commented 1 year ago

@CN120, @adrian-skybaker, @FreshPrinceMayo, @njkoopman, @CCargan, we have completed the agent release, now v3.225.0 is the latest. Could you please confirm the availability of the hashValue property?

FreshPrinceMayo commented 1 year ago

Working for us now

njkoopman commented 1 year ago

It works for us as well, all agents have been updated without problems. Thank for the fix!