Azure Devops Private Feed #203

Closed fabich closed 5 years ago

Which Renovate are you using? CLI, App, or Pro

Renovate app

Which platform are you using? GitHub, GitLab, Bitbucket Azure DevOps

GitHub

What is your question?

How can I connect to a private NuGet feed on Azure DevOps? Using HTTP auth seems not possible and according to https://github.com/renovatebot/renovate/issues/3238 using Nuget.config is not possible either.

You need to add registryurls and hostRules to your config. See https://renovatebot.com/docs/nuget/ for examples

thanks @rarkins Microsoft does not encourage using username and password.

Personal access tokens are a more convenient and secure replacement for Alternate authentication credentials. Please use personal access tokens instead.

@fabich so do they support Bearer authentication instead?

Their docs imply you can use an empty username and personal access token as password:

So, configure hostRules as per the example but try empty string username ("") and token in the password.

I updated the config: { "extends": ["config:base"], "nuget": { "registryUrls": [ "https://api.nuget.org/v3/index.json", "https://pkgs.dev.azure.com/<teamname>/_packaging/<feedname>/nuget/v3/index.json" ] }, "hostRules": [ { "platform": "nuget", "endpoint": "https://pkgs.dev.azure.com/<teamname>/_packaging/<feedname>/nuget/v3/index.json", "username": "", "password": "<base64token>" } ] }

and used the following code to convert access token: var personalaccesstoken = "<accesstoken>"; var base64 = Convert.ToBase64String( System.Text.Encoding.ASCII.GetBytes( string.Format("{0}:{1}", "", personalaccesstoken)));

still getting an error on renovate dashboard: INFO: Failed to look up dependency

Don’t base64 encode yourself. Renovate will handle that. You just need to provide the token in the “password” field.

Don’t base64 encode yourself. Renovate will handle that. You just need to provide the token in the “password” field.

I tried both, same result.

Can you confirm manually that the token works by using one of the examples that Azure gives (in their docs that I linked to earlier)?

Alternatively, is there a simple way to reproduce this issue eg with a test/dummy account?

Hi, @rarkins. I'm jumping in for @fabich. We can confirm that the token works with the example provided in the docs.

I have a dummy NuGet feed hosted on Azure Artifacts ready. Do you have a Microsoft account? I could then grant you access and you can generate your own PAT to reproduce the issue. Or let me know and I can provide you a PAT together with the feed URL.

Here is the debug output from the log:

DEBUG: nuget registry failure: can't get SearchQueryService form https://pkgs.dev.azure.com//_packaging//nuget/v3/index.json { "e": { "name": "HTTPError", "host": "pkgs.dev.azure.com", "hostname": "pkgs.dev.azure.com", "method": "GET", "path": "//_packaging//nuget/v3/index.json", "protocol": "https:", "url": "https://pkgs.dev.azure.com//_packaging//nuget/v3/index.json", "gotOptions": { "path": "//_packaging//nuget/v3/index.json", "protocol": "https:", "slashes": true, "auth": null, "host": "pkgs.dev.azure.com", "port": null, "hostname": "pkgs.dev.azure.com", "hash": null, "search": null, "query": null, "pathname": "//_packaging//nuget/v3/index.json", "href": "https://pkgs.dev.azure.com//_packaging//nuget/v3/index.json", "headers": { "user-agent": "Renovate Bot (GitHub App 2740)", "accept": "application/json", "accept-encoding": "gzip, deflate" }, "hooks": { "beforeError": [], "init": [], "beforeRequest": [], "beforeRedirect": [], "beforeRetry": [], "afterResponse": [] }, "retry": { "methods": {}, "statusCodes": {}, "errorCodes": {} }, "decompress": true, "throwHttpErrors": true, "followRedirect": true, "stream": false, "form": false, "json": true, "cache": false, "useElectronNet": false, "method": "GET" }, "statusCode": 401, "statusMessage": "Unauthorized", "headers": { "content-length": "332", "content-type": "application/json", "p3p": "CP=\"CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT\"", "www-authenticate": "Bearer authorization_uri=https://login.windows.net/37dd6c69-8324-40b3-9a94-643804216ff2, Basic realm=\"https://pkgsprodsu3weu.app.pkgs.visualstudio.com/\", TFS-Federated", "x-tfs-processid": "4f42ebce-ca80-41e7-ac9c-c1b56c8c64b9", "strict-transport-security": "max-age=31536000; includeSubDomains", "activityid": "a1904441-bd19-4507-af7a-f6d82593a789", "x-tfs-session": "a1904441-bd19-4507-af7a-f6d82593a789", "x-vss-e2eid": "a1904441-bd19-4507-af7a-f6d82593a789", "x-tfs-fedauthrealm": "https://pkgsprodsu3weu.app.pkgs.visualstudio.com/", "x-tfs-fedauthissuer": "https://www.visualstudio.com/", "x-vss-authorizationendpoint": "https://.vssps.visualstudio.com/", "x-vss-resourcetenant": "37dd6c69-8324-40b3-9a94-643804216ff2", "x-tfs-serviceerror": "TF400813%3a+Resource+not+available+for+anonymous+access.+Client+authentication+required.", "x-vss-s2stargetservice": "00000030-0000-8888-8000-000000000000/visualstudio.com", "x-tfs-fedauthredirect": "https://spsprodweu4.vssps.visualstudio.com/_signin?realm=pkgs.dev.azure.com&reply_to=https%3A%2F%2Fpkgs.dev.azure.com%2F%2F_packaging%2F%2Fnuget%2Fv3%2Findex.json&redirect=1&aadinmsa=true&hid=6a22ee76-f546-420a-9f6f-09fbae35f513&context=eyJodCI6MiwiaGlkIjoiYTIwZDhiODktMzVlZC00Yjc2LWEwMjMtZmVlZjhkNTk5YzAxIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ud2luZG93cy5uZXQiXX01", "x-content-type-options": "nosniff", "x-msedge-ref": "Ref A: 80DCD2CBB08044CFBDFE67C655F174C8 Ref B: BY3EDGE0106 Ref C: 2019-04-15T11:43:09Z", "date": "Mon, 15 Apr 2019 11:43:08 GMT", "connection": "close" }, "body": "{\"$id\":\"1\",\"innerException\":null,\"message\":\"TF400813: Resource not available for anonymous access. Client authentication required.\",\"typeName\":\"Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server\",\"typeKey\":\"UnauthorizedRequestException\",\"errorCode\":0,\"eventId\":3000}" } }

@mreinli that's great, thanks. My Azure DevOps login is rhys@arkins.net. Can you add me with enough permissions that any PAT of mine should get access? Also please let me know the exact feed to try.

You should have received an invite. As long as the scope of your PAT is including Package Management it should work.

This is the feed URL: https://mr3ch.pkgs.visualstudio.com/_packaging/DevTestFeed/nuget/v3/index.json

Thanks. And is there a single file repo I can create that should trigger the lookup?

First, trying to GET the feed without auth. As expected, I receive a 401 Unauthorized:

{
    "$id": "1",
    "innerException": null,
    "message": "TF400813: Resource not available for anonymous access. Client authentication required.",
    "typeName": "Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server",
    "typeKey": "UnauthorizedRequestException",
    "errorCode": 0,
    "eventId": 3000
}

I then configured Basic Authentication using Postman using empty username and my PAT as password. It works.

Next step will be testing it in Renovate.

First thing I found is that - for now - hostRules must be added to the top level of the renovate config, rather than nested inside the manager name. i.e. like this:

{
  "nuget": {
    "registryUrls": ["https://mr3ch.pkgs.visualstudio.com/_packaging/DevTestFeed/nuget/v3/index.json"]
  },
  "hostRules": [{
    "platform": "nuget",
    "endpoint": "https://mr3ch.pkgs.visualstudio.com",
    "username": "",
    "password": "abc123"
  }]
}

Second problem was that the empty username is somehow not working - I have just committed a patch for that which will be soon released.

Solving those two means we are querying the Azure DevOps URL with authentication and then get new problems:

Our code searches for an exact SearchQueryService type whereas Azure returns SearchQueryService-3.0.0-beta. Should we just look for any value starting with SearchQueryService
The data array referenced here is empty. i.e. no versions

Cc @fgreinacher who helped with the v3 nuget lookup code

Docs imply we should loosen our search query service string matching: https://docs.microsoft.com/en-us/nuget/api/service-index

And empty data array implies there’s simply no result from our package query. Need to double check that the package we are querying matches with what’s in the devops registry.

@rarkins do you have an update on this issue?

It seems like the API from Azure DevOps differs slightly from Nuget's default one.

This works: https://mr3ch.pkgs.visualstudio.com/_packaging/f375f9bd-0033-466e-9b02-82c2dc7c8f87/nuget/v3/query2/?q=Renovate.HelloWorld

This does not: https://mr3ch.pkgs.visualstudio.com/_packaging/f375f9bd-0033-466e-9b02-82c2dc7c8f87/nuget/v3/query2/?q=PackageId:Renovate.HelloWorld

I'm not sure if we can somehow "know" which format to use, or if we need to try both if the more exact PackageId: query fails

This should now be working in the latest version of Renovate. Example renovate.json if using the app:

{
  "nuget": {
    "registryUrls": ["https://mr3ch.pkgs.visualstudio.com/_packaging/DevTestFeed/nuget/v3/index.json"]
  },
  "hostRules": [{
    "platform": "nuget",
    "endpoint": "https://mr3ch.pkgs.visualstudio.com",
    "username": "",
    "encrypted": {
      "password": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="
    }
  }]
}

Or you can put the token/password in directly/unencrypted if it's a private repo and you prefer.

@rarkins thank you very much! We can confirm that it is working now

renovatebot / config-help