[Bug] Azure artifacts authentication

Js-Brecht commented 5 years ago

Describe the issue

Really not sure this is a bug; more like a support question. I am having trouble getting my Azure DevOps Artifacts npm registry working with berry. It works fine in v1, using the token from .npmrc, which is generated by Azure DevOps.

The ":_password" field in the .npmrc is a base64 encoded Personal Access Token generated in Azure DevOps, and according to their documentation, the ":username" and ":email" field mean nothing. Documentation is found here

I have tried generating the auth token through their Artifact "Connect to feed" process, which generates the token for your .npmrc like this

; Treat this auth token like a password. Do not share it with anyone, including Microsoft support. This token expires on or before 10/10/2019.
; begin auth token
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/registry/:username=<username removed.  It was the name of my organization or feed... they are the same>
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/registry/:_password=<password removed>
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/registry/:email=npm requires email to be set but doesn't use the value
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/:username=<username removed>
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/:_password=<password removed>
//pkgs.dev.azure.com/<organization>/_packaging/<artifact feed>/npm/:email=npm requires email to be set but doesn't use the value
; end auth token

I also tried generating my own PAT, and base64 encoding it, for use in that "_password" field.

I used that "_password" field as the npmAuthToken in my .yarnrc.yml, I've tried using it in the npmAuthIdent in combination with the username from above, as <username>:<token>. I've tried combining the <username>:<token> into a base64 encoded string and including that in the npmAuthToken. None of these work. I always get (401) Unauthorized.

To Reproduce

You would need a module published in an Azure DevOps Artifacts registry.

My .yarnrc.yml is below, with the sensitive information redacted.

npmRegistries:
  //pkgs.dev.azure.com/<organization>/_packaging/<azurefeed>/npm/registry:
    npmAlwaysAuth: true
    npmAuthToken: "<_password field from above>"
  //pkgs.dev.azure.com/<organization>/_packaging/<azurefeed>/npm:
    npmAlwaysAuth: true
    npmAuthToken: "<_password field from above>"

npmScopes:
  <org>:
    npmRegistryServer: https://pkgs.dev.azure.com/<organization>/_packaging/<azurefeed>/npm/registry

Environment if relevant (please complete the following information):

OS: Linux
Node version: v12.6.0
Yarn version: v2.0.0

arcanis commented 5 years ago

Can you try using npmAuthIdent instead of a token? Given what Azure asks you to use as configuration, I think they're expecting the token to be sent as a password rather than a token, for some reason.

If it still doesn't work it'd be helpful if you could add some logs near the place that makes the http request to find out whether the server answers with the reason why the request is rejected.

Js-Brecht commented 5 years ago

I did try using npmAuthIdent, with the same "username" and "_password" fields from the .npmrc, in the <username>:<password> format. Still got the error. I actually tried that before I tried using npmAuthToken field. The configuration I left in my original post was the final state it was in before I gave up.

Here's the things I tried:

Generate .npmrc configuration using Azure's "Connect To Feed"
- Using .npmrc fields _password & username in .yarnrc.yml -> npmRegistries:
  - npmAuthToken: <_password>
  - npmAuthToken: <base64 encoded(_password)>
  - npmAuthToken: <username:_password>
  - npmAuthToken: <base64 encoded(username:_password)>
  - npmAuthIdent: <username:_password>
  - npmAuthIdent: <username:<base64 encoded(_password)>>
Generate PAT
- Then I did the same thing as above with the generated PAT as _password

I'm now trying this on a Windows PC. What's interesting is their vsts-npm-auth utility generates an _auth field alone in the .npmrc. It creates an .npmrc config that looks like this:

//pkgs.dev.azure.com/<organization>/_packaging/<azurefeed>/npm/registry/:_authToken=<some ridiculously long token>

That's it. It doesn't have the two different URL's (one with /registry at the end, and one without). Just the one. So I tried using that _auth token in the .yarnrc.yml configuration, as the npmAuthToken field, and was actually able to resolve packages. But still cannot fetch them. I get this:

➤ YN0001: │ HTTPError: <@scope/package-name>@npm:1.2.0: Response code 400 (Authentication information is not given in the correct format. Check the value of Authorization header.)
    at EventEmitter.emitter.on (D:\dev\source\test\yarn-test\.yarn\releases\yarn-berry.js:9730:19)
    at process._tickCallback (internal/process/next_tick.js:68:7)

I tried using that token in the various methods I described above, but the only way it worked was just that token in the npmAuthToken field. The error it would return any other way was this:

➤ BR0027: <@scope/package-name>@unknown can't be resolved to a satisfying range
➤ Errors happened when preparing the environment required to run this command.

I don't know how that vsts-npm-auth utility is generating the token. There's very little documentation on it, and absolutely nothing that details that process, plus it's not open source, so I can't just examine the source.

All I know is that Yarn v1 doesn't have any problem with the .npmrc _auth token, or the _password method either. Neither does npm, or pnpm. Just Yarn/berry. So, it's doing something different, or I have it misconfigured somehow.

Does berry have a built in logging utility, or will I need to do a tcpdump to get the logs you mentioned?

arcanis commented 5 years ago

Very strange ... I think the main difference is that we're not sending the email address as it's unused by all registries I'm aware of 🤔

Does berry have a built in logging utility, or will I need to do a tcpdump to get the logs you mentioned?

The best is to just add a console.log statement before the following line (it should be fairly easy to find it in the standalone bundle):

https://github.com/yarnpkg/berry/blob/master/packages/berry-core/sources/httpUtils.ts#L63

Js-Brecht commented 5 years ago

I put a console.log in there, but it only reported anything during the resolution stage, and those are successful (return code 200). Then it fails during the fetch stage with the 400 error (Authentication information is not given in the correct format. Check the value of Authorization header.), and the console.log doesn't get hit at all. This is on my Windows machine at work.

I will try this again on my Linux box tonight to get some logs for that 401 (Unauthorized) error.

I realized what I've been doing wrong with npmAuthIdent after replicating the process with curl. curl, by default, base64 encodes the string that you pass to the -u option. That made me realize that I've been encoding the field npmAuthIdent wrong this entire time. I have been encoding the PAT, but leaving the rest like username:<encoded PAT>, which looking at now makes no sense. I had also tried encoding the entire string, but only after I had already encoded the PAT... 🙄

So, after encoding <username>:<raw PAT> as base64, and putting that in the npmAuthIdent, I successfully authenticate with the server. So now I don't need the npmAuthToken (which would be too complicated to get for use on my Linux box). However, I am still getting the 400 (Authentication information is not given in the correct format. Check the value of Authorization header.) error. I do think it is expecting an email field, even if it doesn't mean anything.

Here's what I got from curl...

Hit the address that was being used as the target in the HttpUtil.
- Got the JSON content for the package in question

Hit the tarball URL retrieved from the JSON

*   Trying 13.107.42.20...
* TCP_NODELAY set
* Connected to pkgs.dev.azure.com (13.107.42.20) port 443 (#0)
(SSL/TLS handshake omitted for brevity)
* Server auth using Basic with user '<my email address>'
> GET /<organization>/_packaging/<feed>/npm/registry/@<scope>/<package>/-/<package>-1.2.0.tgz HTTP/1.1
> Host: pkgs.dev.azure.com
> Authorization: Basic <base64 encoded <email:personal access token>
> User-Agent: curl/7.55.1
> Accept: */*
(schannel output omitted for brevity)
< HTTP/1.1 302 Found
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: -1
< Location: https://1yovsblobprodeus2184.blob.core.windows.net/<server/relative/path>.blob?<package query>
< P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
< X-TFS-ProcessId: b2b4c5ab-39f3-4afb-abb8-099a15970b28
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< ActivityId: 31953bf3-1506-4218-a267-8131454236f4
< X-TFS-Session: 31953bf3-1506-4218-a267-8131454236f4
< X-VSS-E2EID: 31953bf3-1506-4218-a267-8131454236f4
< X-VSS-UserData: <my user data>
< X-FRAME-OPTIONS: SAMEORIGIN
< Request-Context: appId=cid-v1:aafc8b1a-8b8b-40da-82f7-2e27c159556b
< Access-Control-Expose-Headers: Request-Context
< X-Content-Type-Options: nosniff
< X-MSEdge-Ref: Ref A: A41A8C44161C4BCEA1C27C42B3941EC9 Ref B: HNL01EDGE0213 Ref C: 2019-07-31T23:45:47Z
< Date: Wed, 31 Jul 2019 23:45:47 GMT
< Content-Length: 0
<
* Connection #0 to host pkgs.dev.azure.com left intact

Hit the file Location response, and this is what I get

*   Trying 52.179.144.64...
* TCP_NODELAY set
* Connected to 1yovsblobprodeus2184.blob.core.windows.net (52.179.144.64) port 443 (#0)
(SSL/TLS handshake omitted for brevity)
* Server auth using Basic with user '<my username (email)>'
> GET <server/relative/path>.blob?<package query> HTTP/1.1
> Host: 1yovsblobprodeus2184.blob.core.windows.net
> Authorization: Basic <base64 encoded <email:personal access token>>
> User-Agent: curl/7.55.1
> Accept: */*
>
(schannel output omitted for brevity)
< HTTP/1.1 400 Authentication information is not given in the correct format. Check the value of Authorization header.
< Content-Length: 298
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: 6079e633-601e-0064-19fa-4731fa000000
< Date: Wed, 31 Jul 2019 23:46:23 GMT
<
∩╗┐<?xml version="1.0" encoding="utf-8"?>
<Error><Code>InvalidAuthenticationInfo</Code><Message>Authentication information is not given in the correct format. Check the value of Authorization header.
RequestId:6079e633-601e-0064-19fa-4731fa000000
Time:2019-07-31T23:46:23.6899938Z</Message></Error>* Connection #0 to host 1yovsblobprodeus2184.blob.core.windows.net left intact

Where does an npm server usually expect to see the email address in the request? I can run that through curl, and if I am able to retrieve the tarball, we know that's the problem.

Js-Brecht commented 5 years ago

Nevermind, I figured it out.

I was able to retrieve the file succesfully by sending the authentication headers x-www-urlencoded

Using curl, I used -d with the www-urlencoded authentication information like this:

username=<whatever, doesn't matter>&_password=<raw PAT>

Then the --get option to force it to use the GET verb. If I just do -d, it would try to POST using Content-Type: application/x-www-urlencoded, and the server would not accept it (possible verbs are GET, HEAD, PUT, DELETE). Using the GET verb, with the -d data formatted like above, it works.

curl -v -d "username=alsajskghaskl&_password=<PAT>" "https://1yovsblobprodeus2184.blob.core.windows.net/<server/relative/path>.blob?<crazy long package query, session id, sig, etc...>" --get --output <package name>-1.2.0.tgz

Hope this helps.

arcanis commented 5 years ago

But credentials are expected to be sent through the Authorization header, not the query string, I'm confused 😮

Oh wait I might know where this is coming from - in the v1, when resolving a package, we also were storing the returned archive url. In the v2, we only store the package version, and when we need to fetch the archive we combine the version to the registry according to the following convention: /<name>/-/<name>.tgz (this allows us to improve the UX when working with multiple registries, as switching from a mirror to another will properly fetch the tarballs from the new location). Maybe the tarball urls that Azure is returning don't match this convention and thus can't be accessed this way?

If that's the case we track that in #238 - it should be possible to fix it by storing non-conventional tarball urls within the lockfile, the main issue being that I cannot test it on my side since I don't have access to such environments ... so if someone else could do it it would be awesome 😅

This would however imply that the authentication credentials are somehow stored inside the lockfile when using Azure, which sounds surprising. Can you check?

Js-Brecht commented 5 years ago

Ah, yes, you're correct. It did seem a little odd to me that they would pass credentials that way, but I didn't even think to try it without. 🙄

The tarball naming URL's do match that convention, but that tarball URL redirects to the *.blob, which includes session IDs + various other query parameters (sv, sr, si, sig, spr, se, rscl, rscd). Most of those stay constant per each tarball, but the "rscl" appears to be a session ID, and the "sig" appears to be a type of access token, and they change on each request. I don't think storing that URL will be effective, because I doubt there's a guarantee for how long those parameters will be good.

The only thing they seem to care about when hitting that redirect is whether or not there IS an authorization header. Apparently, everything up to the redirect you get when you hit the tarball URL is authenticated, but the *.blob I get from the tarball URL is anonymous. It simply ignores those query parameters. So the 400 error about checking the Authorization header was just to say "why is there an Authorization header?" 🤣

EDIT: It appears to be the way the redirect is being handled between versions. In v1, the authentication headers must not be sent with the redirected request, while in v2, they are.

Js-Brecht commented 5 years ago

Btw, I can say with certainty that the authentication credentials are not stored in the lockfile when using Azure. At least not with v1. It's just the standard naming convention:

"@<scope>/<package>@^1.2.0":                                                                                                                                                                                                                                            
   version "1.2.0"                                                                                                                                                                                                                                                                
   resolved "https://<registry url>/@<scope>/<package>/-/<package>-<version>.tgz#<hash>"                                                                                      
   integrity <sha1-hash>                                                                                                                                                                                                                                   
   dependencies:                                                                                                                                                                                                                                                       
       "@<scope>/<package>" <version>

I haven't been able to fetch the package successfully with v2 yet, so I can't say what the lockfile looks like for it.

arcanis commented 5 years ago

So just to be clear - the fix would be to do as we do, but if we receive a redirect not send the authentication header after handling the redirect? If so that seems acceptable. Would you be willing to make a PR? The relevant code is here:

https://github.com/yarnpkg/berry/blob/master/packages/berry-core/sources/httpUtils.ts#L63

Js-Brecht commented 5 years ago

Added pull request #329.

arcanis commented 5 years ago

Fixed by #329 🎉

darthtrevino commented 4 years ago

For future readers - tldr: To connect to Azure Artifacts:

npmRegistryServer: "https://your-server"
npmAuthIdent: "base64(your-org:your-pat)"

your-pat is probably encoded as base64 in your ~/.npmrc file. You'll need to decode it first. before re-encoding the key.

tmrclark commented 4 years ago

I was able to get passed authentication, but now I am getting a 404. yarn add @my-scope/my-package@1.0.0 gives this error @my-scope/my-package@npm:1.0.0: Response code 404 (Not Found)

The following returns 200 curl GET 'https://pkgs.dev.azure.com/<my-org>/_packaging/<my-feed>/npm/registry/<my-package>' --header 'Authorization: base64(my-org:my-pat)

Not sure if this is related to Yarn or Artifacts. Thought I would share this here in case somebody is having the same issue. I have a question out on stack overflow too

pats commented 3 years ago

Problem still exists, lacks of solution. Are we able to provide more details, any kind of yarnrc.yml as example?

rachelslurs commented 2 years ago

For future readers - tldr: To connect to Azure Artifacts:
npmRegistryServer: "https://your-server"
npmAuthIdent: "base64(your-org:your-pat)"
your-pat is probably encoded as base64 in your ~/.npmrc file. You'll need to decode it first. before re-encoding the key.

THANK YOU @darthtrevino ! This just worked for me using github registry as well. Finally!

jpzwarte commented 2 years ago

To clarify this even more:

Follow the "Connect to feed", "NPM" instructions to generate a PAT token

But in step 3.2, type:

<organisation>:<PAT>

Then your .yarnrc.yml should look like this (replace <...> with your values):

npmRegistries:
  //pkgs.dev.azure.com/<organisation>/_packaging/<azurefeed>/npm/registry:
    npmAlwaysAuth: true
    npmAuthIdent: <base64 output from above>

npmScopes:
 <azurefeed>:
    npmRegistryServer: "https://pkgs.dev.azure.com/<organisation>/_packaging/<azurefeed>/npm/registry"

After you have this setup, yarn npm info @scope/@mypackage should work!

I've tested this with yarn berry/v3.

tomschut commented 1 year ago

for us the base64 encoding did not work, but having the \<org>:\<pat> in plain form in the npmAuthIdent did

cherealnice commented 1 year ago

Then your .yarnrc.yml should look like this (replace <...> with your values):

npmRegistries:
  //pkgs.dev.azure.com/<organisation>/_packaging/<azurefeed>/npm/registry:
    npmAlwaysAuth: true
    npmAuthIdent: <base64 output from above>

npmScopes:
 <azurefeed>:
    npmRegistryServer: "https://pkgs.dev.azure.com/<organisation>/_packaging/<azurefeed>/npm/registry"

In case anyone else has a similar setup to me, we run all modules through our azure registry, so our shared .yarnrc.yml looked like this:

npmRegistryServer: "https://pkgs.dev.azure.com/<org>/_packaging/<feed>/npm/registry/"

npmRegistries:
  //pkgs.dev.azure.com/<org>/_packaging/<feed>/npm/registry/:
    npmAlwaysAuth: true
    npmAuthIdent: "<org>:${AZURE_ARTIFACTS_TOKEN}"

And then each dev can have their own $AZURE_ARTIFACTS_TOKEN in their PATH, and it's easy to set the env var in a build

hiraldesai commented 1 year ago

@cherealnice I came across similar issue recently. Our dev environment uses npm install -g ado-auth ado-auth -d to automatically setup tokens based on the .npmrc and .yarnrc.yml files. But the same cannot be done on the ADO build server because ado-auth requires interactive auth. I solved it by doing the following:

Content of the yarnrc.yml like this (only used for local)

nodeLinker: node-modules

npmRegistryServer: "https://your-org.pkgs.visualstudio.com/_packaging/npm-mirror/npm/registry/"

yarnPath: .yarn/releases/yarn-3.6.1.cjs

Create a new yarnrc-ci.yml file with this content

nodeLinker: node-modules

npmRegistryServer: "https://domoreexp.pkgs.visualstudio.com/_packaging/npm-mirror/npm/registry/"

npmRegistries:
  //my-org.pkgs.visualstudio.com/_packaging/npm-mirror/npm/:
    npmAlwaysAuth: true
    npmAuthToken: ${SYSTEM_ACCESSTOKEN}
  //my-org.pkgs.visualstudio.com/_packaging/npm-mirror/npm/registry/:
    npmAlwaysAuth: true
    npmAuthToken: ${SYSTEM_ACCESSTOKEN}

yarnPath: .yarn/releases/yarn-3.6.1.cjs

Set environment variable for the yarnrc file in the build script:

  - pwsh: |
      echo "##vso[task.setvariable variable=YARN_RC_FILENAME].yarnrc-ci.yml"
    displayName: "Set yarnrc file"

Use the yarnrc file name and access token environment variable in your build script for all the tasks that depend on yarn. Please note that System.AccessToken is an environment variable that ADO provides to entire pipeline which is passed as SYSTEM_ACCESSTOKEN to the yarnrc-ci.yml file below.

  - task: Yarn@3
    displayName: "Installing Yarn packages"
    inputs:
      arguments: "--immutable"
    env:
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
      YARN_RC_FILENAME: $(YARN_RC_FILENAME)

konclave commented 6 months ago

It's possible to make the authentication without creating the PAT, but with token generated by npmAuthentication@0:

It's a bit hacky, but it works – we read the JWT token generated by npmAuthentication@0 from .npmrc and set it to the .yarnrc.yml as npmAuthToken

# without registry set npmAthetication won't add a token
- script: |
    echo "registry=https://pkgs.dev.azure.com/<your-group>/_packaging/SIP/npm/registry" >> .npmrc
    echo "always-auth=true" >> .npmrc
- task: npmAuthenticate@0
  inputs:
    workingFile: .npmrc
# replacing npmAuthIdent (which is base64(user:password)) with npmAuthToken (JWT token generated by npmAuthentication)
- script: |
    yarn config unset npmScopes.<your-organisation-scope>.npmAuthIdent
    yarn config set npmScopes.<your-organisation-scope>.npmAuthToken $(awk -F "=" '/_authToken/ {print $2}' .npmrc)

yarnpkg / berry

[Bug] Azure artifacts authentication #316