Download artifacts using the new azure API

epananth commented 3 years ago

[ ] This issue is blocking
[ ] This issue is causing unreasonable pain

Previously we downloaded one asset at a time using this API (4.1 preview version) https://dev.azure.com/dnceng//_apis/resources/Containers/{containerId}?itemPath=/BlobArtifacts/MergedManifest.xml&isShallow=true&api-version=4.1-preview.4

Where we use it -> https://github.com/dotnet/arcade/blob/0916e2d4f072d0a87db15803ce5572ff687354d4/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs#L885

There is a better way to do this

https://dev.azure.com/dnceng/_apis/resources/Containers/{containerId}?itemPath=PackageArtifacts

This will get the list of all the packages and we do not have to construct the url to download the asset anymore

Eg: 

{
    "count": 2,
    "value": [
        {
            "containerId": 6973879,
            "scopeIdentifier": "",
            "path": "PackageArtifacts",
            "itemType": "folder",
            "status": "created",
            "dateCreated": "2021-05-05T01:02:21.667Z",
            "dateLastModified": "2021-05-05T01:02:21.667Z",
            "createdBy": "4c028388-2440-4c98-8b52-ed039803d8c5",
            "lastModifiedBy": "4c028388-2440-4c98-8b52-ed039803d8c5",
            "itemLocation": "https://dev.azure.com/dnceng/_apis/resources/Containers/6973879?itemPath=PackageArtifacts&metadata=True",
            "contentLocation": "https://dev.azure.com/dnceng/_apis/resources/Containers/6973879?itemPath=PackageArtifacts",
            "contentId": ""
        },
        {
            "containerId": 6973879,
            "scopeIdentifier": "",
            "path": "PackageArtifacts/Microsoft.SourceBuild.Intermediate.diagnostics.5.0.0-preview.21254.1.nupkg",
            "itemType": "file",
            "status": "created",
            "fileLength": 97851,
            "fileEncoding": 1,
            "fileType": 1,
            "dateCreated": "2021-05-05T01:02:21.663Z",
            "dateLastModified": "2021-05-05T01:02:21.663Z",
            "createdBy": "4c028388-2440-4c98-8b52-ed039803d8c5",
            "lastModifiedBy": "4c028388-2440-4c98-8b52-ed039803d8c5",
            "itemLocation": "https://dev.azure.com/dnceng/_apis/resources/Containers/6973879?itemPath=PackageArtifacts%2FMicrosoft.SourceBuild.Intermediate.diagnostics.5.0.0-preview.21254.1.nupkg&metadata=True",
            "contentLocation": "https://dev.azure.com/dnceng/_apis/resources/Containers/6973879?itemPath=PackageArtifacts%2FMicrosoft.SourceBuild.Intermediate.diagnostics.5.0.0-preview.21254.1.nupkg",
            "artifactId": artifactId,
            "contentId": ""
        }
    ]
}

cc: @hoyosjs @mmitche

hoyosjs commented 3 years ago

Additionally, it gives you a way to check the size of the downloaded payload for a simple smoke test, so I really like this option for reliability.

MattGal commented 3 years ago

[Async Triage]: I agree the improvements seem compelling, not sure what epic it belongs in though. Perhaps dotnet/arcade-services#2447 , "Improve Release process" ? (accidentally put it there while looking at lists)

epananth commented 3 years ago

Probably can go in Unify and improve build publishing process across (6.x, 5.x and 3.x)

epananth commented 3 years ago

Cos that epic deals with removing v1 and v2 publishing. Also can update stuff for V3 publishing also

markwilkie commented 3 years ago

Agreed - done

mmitche commented 2 years ago

@epananth Did you do this work when you did the streaming publishing work?

epananth commented 1 year ago

sorry missed this :/ this is not completed.

dotnet / dnceng

Download artifacts using the new azure API #1268