download universal artifacts

cataggar commented 9 months ago

Is it possible to download universal artifacts? See MS internal post, but I'm looking for an alternative to

az artifacts universal download \
    --path "." \
    --scope "project" \
    --feed "my-azure-feed \
    --name "horust" \
    --version "0.1.7";

It looks like I'm not the only one looking for an az artifacts universal download alternative: https://developercommunity.visualstudio.com/t/download-universal-package-rest-api/616820

johnbatty commented 9 months ago

Not sure... I'll do some investigation.

johnbatty commented 9 months ago

There are quite a few articles saying that you can't download universal artifacts with a simple HTTP call, e.g.

I did find this blog post which claims to know how to do it, and promises to explain all in a subsequent post but unfortunately hasn't written that one!

https://navrockclimber.github.io/html/2021/09/10/downloading-devops-artifact.html

I'll try to investigate what artifacttool does.

cataggar commented 9 months ago

I investigated what artifacttool does. I installed Azure CLI and mitmproxy in CBL-Mariner Linux. I added the proxy certificate to the list of trusted certificates following these instructions.

openssl verify ~/.mitmproxy/mitmproxy-ca.pem
sudo cp ~/.mitmproxy/mitmproxy-ca.pem /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
openssl verify ~/.mitmproxy/mitmproxy-ca.pem

After starting mitmproxy or I prefer mitmweb, setting HTTPS_PROXY allows the az artifacts universal download HTTP calls to be captured.

export HTTPS_PROXY=http://127.0.0.1:8080

If using the azure-devops-python-api, for the TLS connection to work, REQUESTS_CA_BUNDLE also has to be set:

export REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt

I then wrote and ran this python get_package.py:

from azure.devops.connection import Connection
from msrest.authentication import BasicAuthentication
import pprint
from beeprint import pp

personal_access_token = ''
organization_url = 'https://dev.azure.com/myorg'
project = 'myproj'
feed_id = 'myfeed'
package_name = 'mypackage'
package_version = '0.1.7'

credentials = BasicAuthentication('', personal_access_token)
connection = Connection(base_url=organization_url, creds=credentials)

upack = connection.clients.get_upack_api_client()
package_version = upack.get_package_version(feed_id, package_name, package_version, project)
pp(package_version)

packaging = connection.clients.get_upack_packaging_client()
package = packaging.get_package_metadata(feed_id, package_name, package_version, project)
pp(package)

Notice that there are two different clients. The second one is what is needed for downloading a universal package. I did not find it in the API specs. The first client is from this spec:

vsts-rest-api-specs\specification\artifactsPackageTypes\7.1\universal.json "x-ms-vss-method": "GetPackageVersion" GET "/{organization}/{project}/_apis/packaging/feeds/{feedId}/upack/packages/{packageName}/versions/{packageVersion}"

The HTTP call for get_package_metadata is: GET "/{organization}/{project}/_packaging/{feedId}/upack/packages/{packageName}/versions/{packageVersion}" The URL is very similar, but no _apis or feeds, and a different json response:

{
    "manifestId": "8BE335749F6313AC7BB4EFCC8025573BE63AEE791836D0B922D73972B1F9A33A01",
    "packageSize": 5464585,
    "superRootId": "DDEFB81F13FAE572147ECCD659A1969D42EC3B157C777C3CE96BB84AB62A8C5102",
    "version": "0.1.7"
}

After this, there are a couple of service lookups, a package manifest get, a call to get a list of chunks per file in the package manifest, then calls to get all the chunks.

krosk commented 6 months ago

@cataggar Interesting! And after getting the chunks, do you know the logic to reconstruct the files?

cataggar commented 2 months ago

I booked 3 days in September to hack on this.

microsoft / azure-devops-rust-api

download universal artifacts #330