Error Unhandled: unexpected end of file : DownloadReleaseArtifact not working in Deployment group job

[aadiver]

I've the following problem when using Download Release Artifact:

2021-07-16T09:43:18.5656441Z ##[section]Starting: Download Release Artifact app-release-TST.apk 2021-07-16T09:43:18.5744895Z ============================================================================== 2021-07-16T09:43:18.5745381Z Task : Download Release Artifact 2021-07-16T09:43:18.5745914Z Description : Let's you download a Release Artifacts that was uploaded to the Release Logs in a previous step 2021-07-16T09:43:18.5746559Z Version : 1.1.7 2021-07-16T09:43:18.5746871Z Author : Benjamin Huser 2021-07-16T09:43:18.5747163Z Help : 2021-07-16T09:43:18.5747537Z ============================================================================== 2021-07-16T09:43:22.3587309Z Will store logs in C:\azagent\A1_work\r49\a\DownloadReleaseArtifactTask 2021-07-16T09:43:22.3590284Z Looking for Artifact with the name app-release-TST.apk 2021-07-16T09:43:22.3591374Z Matching Artifact will be copied to D:\DATA\TST\A1770-id3\APK 2021-07-16T09:43:22.3592562Z No existing Logs found - will initiating download... 2021-07-16T09:43:22.3603483Z Will get logs for current Relese 2021-07-16T09:43:22.3605050Z Organization: https://dev.azure.com/INFRABEL/ 2021-07-16T09:43:22.3605961Z Team Project ID: 326893eb-f65e-4d0a-99da-d8152a57f0ba 2021-07-16T09:43:22.3606627Z Release ID: 562 2021-07-16T09:43:23.9919236Z ##[error]Unhandled: unexpected end of file 2021-07-16T09:43:24.0368171Z ##[section]Finishing: Download Release Artifact app-release-TST.apk

Is the problem maybe related because the agents "Agent job" and "Deployment group job" are both writing and reading from the same release log. So the second agent "Deployment group job" is trying to read the release log but it can't download it yet because this second agent is using the same release log as the first agent "Agent job" and it is not yet free?

[huserben]

Hi @aadiver

is this the same problem as you've mentioned in #9?

For my clarification, the Agent Job and Deployment job are not executed in parallel right, but in sequence? If so there should not be a problem.

What I'm starting to wonder, perhaps the Deployment Group Job itself is an issue - would you be able to try to run the Download Task in an Agent Job and see whether this works? If yes then there is probably a difference between Agent and Deployment Group Job that ain't handled yet.

[aadiver]

I've created finally another issue because I think it is another issue then MatthiasJentsch because he has the problem "No Artifacts found - will fail task" but in my case the artifact is found but it can't download it.

Agent Job and Deployment job are running in sequence.

When executing the "Download Release Artifact" in another Agent Job it doesn't work also:

The "Download Release Artifact" provides the following output: 2021-07-17T06:39:50.5756095Z ##[section]Starting: Download Release Artifact app-release-TST.apk 2021-07-17T06:39:50.5948430Z ============================================================================== 2021-07-17T06:39:50.5948803Z Task : Download Release Artifact 2021-07-17T06:39:50.5949161Z Description : Let's you download a Release Artifacts that was uploaded to the Release Logs in a previous step 2021-07-17T06:39:50.5949479Z Version : 1.1.7 2021-07-17T06:39:50.5949715Z Author : Benjamin Huser 2021-07-17T06:39:50.5949940Z Help : 2021-07-17T06:39:50.5950185Z ============================================================================== 2021-07-17T06:39:51.3404492Z Will store logs in D:\a\r1\a\DownloadReleaseArtifactTask 2021-07-17T06:39:51.3406863Z Looking for Artifact with the name app-release-TST.apk 2021-07-17T06:39:51.3407353Z Matching Artifact will be copied to D:\DATA\TST\A1770-id3\APK 2021-07-17T06:39:51.3408909Z No existing Logs found - will initiating download... 2021-07-17T06:39:51.3414596Z Will get logs for current Relese 2021-07-17T06:39:51.3415000Z Organization: https://dev.azure.com/INFRABEL/ 2021-07-17T06:39:51.3415814Z Team Project ID: 326893eb-f65e-4d0a-99da-d8152a57f0ba 2021-07-17T06:39:51.3416472Z Release ID: 564 2021-07-17T06:39:52.7651391Z ##[error]Unhandled: unexpected end of file 2021-07-17T06:39:52.7749876Z ##[section]Finishing: Download Release Artifact app-release-TST.apk

The "Upload Release Artifact" is doing it works correctly:

Some extra information: When executing the following in the browser it can download the logs: https://vsrm.dev.azure.com/infrabel/326893eb-f65e-4d0a-99da-d8152a57f0ba/_apis/release/releases/555/logs?api-version=6.0-preview.2

So I'm not sure how you are downloading the log file?

[huserben]

Ok, thanks for the clarification and investigation.

If it's not the deployment job, my hunch would be that there might be something blocking the download, like some proxy or firewall within a corporate network. Downloading via the url was working from within the agent you run the jobs?

When you download the logs, is the file you're trying to fetch is actually correct? Like can you open/unzip it and verify that it was properly uploaded (maybe the issue is already happening during uploading but the task doesn't fail...).

In order to download the logs, the following code is used (see also https://github.com/huserben/PublishReleaseArtifactsExtension/blob/master/DownloadReleaseArtifactTask/index.ts#L149)

var authHandler = devopsapi.getHandlerFromToken(token);
   var connection: devopsapi.WebApi = new devopsapi.WebApi(organizationUrl, authHandler);
   var releaseapi = await connection.getReleaseApi();
   var logs = await releaseapi.getLogs(teamProject, releaseId);

This is making use of the official api from Microsoft which is why I have to assume it's working correctly. It's just strange that then it works via browser...

If it really doesn't work you could probably use a simple script, e.g. in Powershell, as a workaround and download it via the url instead - not perfect but might solve the problem for you.

[aadiver]

When downloading with a powershell script I get some more information:

$strURL='https://vsrm.dev.azure.com/infrabel/326893eb-f65e-4d0a-99da-d8152a57f0ba/_apis/release/releases/555/logs?api-version=6.0-preview.2'
$filePath="$(System.DefaultWorkingDirectory)"
$fileName=$filePath+ '\tasklog.log'
$base64auth=[Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f"","$PATtoken")))
$resLog=Invoke-RestMethod -Method Get -ContentType "text/plain" -Uri $strURL -Headers @{Authorization=("Basic {0}"-f$base64auth)}
Write-Host $resLog
Out-File -FilePath $filename -InputObject $resLog -Encoding ascii

OutputPowerShellScript.txt

[huserben]

Hi @aadiver

thanks for further checking this. However I think there might be a problem with the powershell script. At least from the logs it looks like the authentication didn't work and you got redirected to a login page instead of fetching any data.

It looks like you ran the script on the build machine - does the pwsh code work on your local machine? Are you sure the PAT is set correctly when running it via the pipeline?

[aadiver]

I've investigated this further on and can let the PowerShell script running successful. I've created a personal access token and created a deployment group locally on my machine. (I'm working in corporate environment with firewalls, proxies, ...). So I'll try first to download to my local workstation.

$strURL='https://vsrm.dev.azure.com/infrabel/326893eb-f65e-4d0a-99da-d8152a57f0ba/_apis/release/releases/555/logs?api-version=6.0-preview.2'
$base64auth=[Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "","<Fill in your personal access token>")))
Invoke-RestMethod -Method Get -ContentType "text/plain" -Uri $strURL -Headers @{Authorization=("Basic {0}"-f$base64auth)} -OutFile "c:\Temp\release555.zip"

With this PowerShell script I'm able to download the file. In the previous post I didn't provide a personal access token. I thought that the variable $PATtoken was filled in automatically.

When using your component in the same deployment group job I get now the following different error:

2021-07-19T12:19:23.0037359Z ##[section]Starting: Download Release Artifact app-release-TST.apk 2021-07-19T12:19:23.0289000Z ============================================================================== 2021-07-19T12:19:23.0289490Z Task : Download Release Artifact 2021-07-19T12:19:23.0289994Z Description : Let's you download a Release Artifacts that was uploaded to the Release Logs in a previous step 2021-07-19T12:19:23.0290489Z Version : 1.1.7 2021-07-19T12:19:23.0290757Z Author : Benjamin Huser 2021-07-19T12:19:23.0291022Z Help : 2021-07-19T12:19:23.0291399Z ============================================================================== 2021-07-19T12:19:27.8081829Z Will store logs in C:\azagent\A10_work\r1\a\DownloadReleaseArtifactTask 2021-07-19T12:19:27.8086301Z Looking for Artifact with the name app-release-TST.apk 2021-07-19T12:19:27.8087527Z Matching Artifact will be copied to C:\Temp\app-release-TST.apk 2021-07-19T12:19:27.8214675Z No existing Logs found - will initiating download... 2021-07-19T12:19:27.8215480Z Will get logs for current Relese 2021-07-19T12:19:27.8215896Z Organization: https://dev.azure.com/INFRABEL/ 2021-07-19T12:19:27.8216349Z Team Project ID: 326893eb-f65e-4d0a-99da-d8152a57f0ba 2021-07-19T12:19:27.8216737Z Release ID: 593 2021-07-19T12:19:48.9180095Z ##[error]connect ETIMEDOUT 13.107.42.20:443 2021-07-19T12:19:48.9257330Z ##[section]Finishing: Download Release Artifact app-release-TST.apk

Maybe my personal access token has now all the rights but your component is using the SYSTEM_ACCESSTOKEN var token: string = ${process.env["SYSTEM_ACCESSTOKEN"]}; I don't know what is the difference between a PAT token and a SYSTEM_ACCESSTOKEN. Maybe SYSTEM_ACCESSTOKEN doesn't have enough privileges?

[aadiver]

I've tested the PowerShell script also with the SYSTEM_ACCESSTOKEN (you need to use Bearer in place of Basic when using PAT) and this works also.

$strURL='https://vsrm.dev.azure.com/organization/326893eb-f65e-4d0a-99da-d8152a57f0ba/_apis/release/releases/555/logs?api-version=6.0-preview.2'
Invoke-RestMethod -Method Get -ContentType "text/plain" -Uri $strURL -Headers @{Authorization=("Bearer $env:SYSTEM_ACCESSTOKEN")} -OutFile "c:\Temp\release555systemaccesstoken.zip"

With the "Download Release Artifact" task it doesn't work:

[huserben]

Thanks for further investigating.

This is strange, not entirely sure why the task then fails. the Timeout error might be related to something that is blocking the request.

I checked now briefly and saw that apparently others face the same/similar issue: https://github.com/microsoft/azure-devops-node-api/issues/425

Not sure what is causing it, my gut would have told me it's proxy/firewall related but if the powershell script works it's probably not that...

As I'm uncertain how to deal with it, maybe it could make sense to extend your powershell script to be a bit more generic and use that instead to download & extract the files?

[aadiver]

I've added the following working PowerShell script:

$organizationUrl = $env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI
$teamProject = $env:SYSTEM_TEAMPROJECTID
$token = $env:SYSTEM_ACCESSTOKEN
$releaseId = $env:RELEASE_RELEASEID
$strURL="https://vsrm.dev.azure.com/infrabel/${teamProject}/_apis/release/releases/${releaseId}/logs?api-version=6.0-preview.2"

Write-Host 'Will get logs for current Release'
Write-Host "Organization: ${organizationUrl}"
Write-Host "Team Project ID: ${teamProject}"
Write-Host "Release ID: ${releaseId}"
Write-Host "strURL: ${strURL}"

Invoke-RestMethod -Method Get -ContentType "text/plain" -Uri $strURL -Headers @{Authorization=("Bearer $env:SYSTEM_ACCESSTOKEN")} -OutFile "c:\Temp\release${releaseId}.zip"

With this PowerShell script I'm able to download the release log. But your component is a lot smarter. I think it is unzipping the log file and extracting the wanted artifact? I'm not sure what your component is doing extra because I didn't see it yet working?

Is it possible to debug your component? Can you attach a debugger to a running pipeline? Or can you add just add your code to a task which compiles the typescript to JavaScript automatically and executes the code?

[huserben]

But your component is a lot smarter

Well there is not that much more to it - it's really just the extracting and copying in the right place I believe (I'm actually not needing to use the extension anymore...).

Is it possible to debug your component? Can you attach a debugger to a running pipeline?

No not really, unless you want to start debugging on the build agent itself and it's probably not ideal :-/

Or can you add just add your code to a task which compiles the typescript to JavaScript automatically and executes the code?

Not sure I get that. The task is already in Javascript when it's being executed on the build agent.

[aadiver]

I've tested this further. When I want to use the server from the company (a deployment group job on a company server) I've to specify the Proxy parameter for the Invoke-RestMethod command, otherwise it won't work. Is there a possibility to specify a proxy in your component?

Working powershell script:

$organizationUrl = $env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI
$teamProject = $env:SYSTEM_TEAMPROJECTID
$token = $env:SYSTEM_ACCESSTOKEN
$releaseId = $env:RELEASE_RELEASEID
$strURL="https://vsrm.dev.azure.com/infrabel/${teamProject}/_apis/release/releases/${releaseId}/logs?api-version=6.0-preview.2"

Write-Host 'Will get logs for current Release'
Write-Host "Organization: ${organizationUrl}"
Write-Host "Team Project ID: ${teamProject}"
Write-Host "Release ID: ${releaseId}"
Write-Host "strURL: ${strURL}"

Invoke-RestMethod -Method Get -ContentType "text/plain" -Uri $strURL **-Proxy http://proxyserver.xx.be:80** -Headers @{Authorization=("Bearer $env:SYSTEM_ACCESSTOKEN")} -OutFile "$(targetDirectory)\release${releaseId}.zip"

[huserben]

@aadiver thanks for checking this out in more detail.

Normally you set the proxy when you configure the the agent on the server. I assume you registered your server by copying the script as described in https://docs.microsoft.com/en-us/azure/devops/pipelines/release/deployment-groups/?view=azure-devops. I believe you could extend it with a --proxyurl http://proxyserver.xx.be:80 like described here. Even though it talks about pipeline agents, I think it will work also for the deployment group agent.

That way everything you execute should know about the agent. Could you try that out and see whether that helps?