Support authenticating pub against 3rd-party servers

kevmoo commented 8 years ago

We've discussed something like pub authenticate example.com

This would allow private pub repos to have authenticated access.

We'd also want authentication to be supported for pub get/update

CC @computmaxer

donny-dont commented 8 years ago

Any idea what authentication schemes would be supported?

I'm moving speakeasy into its own org, https://github.com/speakeasy-pub/speakeasy, and would be very interested in this conversation.

I've had success at work with https://github.com/rlidwka/sinopia for NPM and that has multiple authentication backends. Open to doing something similar with speakeasy.

computmaxer commented 8 years ago

My understanding is that they would extend the same OAuth2 functionality that already exists for the pub publish command.

nex3 commented 8 years ago

That's right. The plan is to add a pub authenticate HOSTNAME command that will send the user through the OAuth2 authentication process, eventually providing a token that the given host can use to verify the user's google account. This token will be stored in the user's pub cache, and sent along with any requests to that server in the course of version resolution.

dynaxis commented 8 years ago

For small organizations like us, it's really beneficial to use Google's identity management. And I'll be happy with it. But just of out curiosity and for the larger organizations than us requiring use of their own id management, does it require much effort to implement support for oauth2 id service other than Google's?

nex3 commented 8 years ago

There are some technical constraints that make it difficult. OAuth2 requires that the client—in this case pub—have a client ID and secret provided to it out-of-band by the identity provider. We already have such an ID and secret for Google built in to pub, but getting it out-of-band for arbitrary identity servers is harder. Not impossible, but not necessary for what we're doing here either.

bjornm commented 8 years ago

Maybe a first step could be to enable authentication for pub get using the same authentication mechanism that is already used for pub publish? E.g.

export PUB_GET_AUTHENTICATION_ENABLED=true
pub get
# will use ~/.pub-cache/credentials.json just like pub publish

We run our own dart pub package server and we are fine with using google oauth for pub publish. Our server verifies the authorization token against Google. The returned user is checked against a list of allowed uploaders. Adding pub get authentication would give us what we need to protect the repository from unauthorized users.

nex3 commented 8 years ago

We can't send users' pub.dartlang.org auth tokens to other servers; it would allow any server that they download packages from to spoof them and upload packages on their behalf.

bjornm commented 8 years ago

What if pub get tokens could be restricted to PUB_HOSTED_URL so it is only on intent that it is used?

Similar to how pub publish behaves right now. It is sending the pub.dartlang.org auto tokens to our server since we override it:

export PUB_HOSTED_URL=https://...

That also opens up for spoofing in a sense. Not that it matters to us because we are completely aware of this and trust our own pub package server.

nex3 commented 8 years ago

I'm not comfortable assuming that all users will be completely aware of what they're doing in that case. The only way I'm willing to do this is if each hosted URL has its own credentials.

bjornm commented 8 years ago

Ok, makes sense.

trentgrover-wf commented 8 years ago

any movement on this @nex3 @kevmoo ?

nex3 commented 8 years ago

Not yet. I'm hoping to budget time for it next quarter.

computmaxer commented 8 years ago

Any update on this?

kevmoo commented 7 years ago

We still want to do this – but we likely won't get to it until early 2017.

thosakwe commented 7 years ago

I'm willing to help out with this.

kevmoo commented 7 years ago

We'd love help here – but it's critical we have a design that's approved upfront.

If you're interested, @thosakwe ...

thosakwe commented 7 years ago

I have a fork of pub called "re:pub" that supports basic auth, in addition to Google OAuth2.

Perhaps users could store their basic auth credentials in a file, which could be read as follows:

# .pub-auth.yaml
username: jdoe1
password: foobarbaz24

And then, run pub with an argument to point to the file. This is assuming the user has set a "PUB_HOSTED_URL."

pub publish --auth-file .pub-auth.yaml

And then, perhaps users could optionally specify a host in the YAML file, so they wouldn't have to set an environment variable every time.

This design would make it possible to use dependencies from public pub, as well as publishing to a private host.

However, there would need to be some sort of distinction made between private and normal dependencies. I think that in the "pubspec", private dependencies would have an extra attribute added, whereas normal ones would remain the same.

# pubspec.yaml
dependencies:
  string_scanner: ^1.0.0 # normal
  in_house_package:
    hostname: https://pub.my-private-company.com
    version: ^0.3.5

Let me know what you think!

nex3 commented 7 years ago

I'm not a big fan of adding a bunch of extra configuration files. We already have .pub-cache/credentials.json as a global config file, so I think any new configuration should go in there. I'd like the user's interaction with it to all go through the pub executable—for example, pub auth add http://example.org could authenticate the user with that domain.

We already have means of depending on hosted packages from arbitrary domains, and of specifying what domain a package should be published to. I don't think we need to add extra functionality for that.

donny-dont commented 7 years ago

Currently speakeasy does caching of packages from pub.dartlang.org.

NPM has scoping which lets you easily target multiple registries if you need to. Not sure how pub would do that at the moment.

thosakwe commented 7 years ago

I'm not sure how to handle scoping, either but...

In response to @nex3: On second thought, adding additional credentials to the .pub-cache/credentials.json is better all-around.

I feel that it might be easier to support Basic Auth for third-party hosts. Otherwise, every third-party host would have to implement the OAuth2 spec, and Pub would have to remember a new Client ID and Client Secret for each host.

Perhaps Pub could support an HTTP header, X-Pub-Auth-Required , or something similar. If the server returns a 403 on a request, and X-Pub-Auth-Required is set to basic, then Pub would prompt the user for a username and password, and retry the request. If another 403 is thrown, then Pub would abort the request. In this case, the server could send another HTTP header to describe the authentication error, X-Pub-Auth-Error: Incorrect password. Otherwise, a default error message could be shown.

Just a thought. I do think this would work, though.

nex3 commented 7 years ago

I have usability concerns about supporting basic auth. Either we'd have to make users authenticate for every invocation of pub, which is a very substantial usability burden; or we'd need to store the user's raw credentials in credentials.json, which isn't a good security practice and may not be clear to users, since we don't store pub.dartlang.org credentials that way. One of the major benefits of a system like OAuth2 is that the stored information doesn't make the user's password visible, and I'd be unhappy to give that up.

thosakwe commented 7 years ago

Ah, I understand. So in that case, would it be difficult to store a client ID/secret, as well as an access token for each third-party host?

SCKelemen commented 7 years ago

Is any work currently being done on this? Could someone recommend a solution for private packaging?

thosakwe commented 7 years ago

Currently the best solution is to use private Git repositories.

travissanderson-wf commented 7 years ago

Private git repositories work pretty well, the primary shortcoming is that pub can't do version range resolution with those. Until auth is added, we are running our private pub server with local network access only. We have a huge selection of private libraries so range resolution is a must at this point.

SCKelemen commented 7 years ago

I have a private repo for my components. Is it possible to use pub to retrieve it though? If not, how would I references a directory or repo?

thosakwe commented 7 years ago

dependencies:
  my_package:
    git: git://url

You can also use an HTTPS URL there, i.e. Github

travissanderson-wf commented 7 years ago

Note that this currently only works for one package per repo. I think there is an improvement coming in the next SDK to let you define a sub-directory.

travissanderson-wf commented 7 years ago

Here's the docs for the current method of declaring git dependencies. This is the improvement coming in Dart 1.25 to allow depending on specific directories in a git repo:

Git dependencies may now include a path parameter, indicating that the package exists in a subdirectory of the Git repository

SCKelemen commented 7 years ago

@travissanderson-wf @thosakwe Thanks guys, I trolled through a lot of documentation, but I didn't see this. I appreciate you steering me in the right direction.

computmaxer commented 6 years ago

This issue still causes Workiva pain on a regular basis. Without OAuth2 support, we're forced to whitelist access to our pub server by IP address. This involves many pain points:

We have a fair number of remote workers, or developers who work from home occasionally. They are forced to VPN into our network in order to run pub get. This is the only part of our entire development environment (including python, go, and java package repositories) that requires developers to VPN in.
Automated CI build systems have to manually be added to the firewall whitelist. This causes pain as the devops teams make changes to the build system's infrastructure.
We have an annual internal developer's conference at a convention center with ~250 developers in attendance. Developers who are usually in an office have to worry about requesting VPN access before the week-long event. IT has to worry about unusually high load on the VPN system.
Whitelisting IP addresses is a poor mechanism to secure business-critical source code.

Would be happy to discuss this issue further if needed! :)

nex3 commented 6 years ago

This is still something I think would be beneficial, but we haven't had any resources to allocate towards it. If someone externally wanted to work on it I'd be happy to review.

isoos commented 6 years ago

@nex3: do you have some pointers on what the scope and the extent of such work could be? A friend have complained to me about the inability of using a private pub server with configurable auth (which contributed to their decision to go ahead with the npm world instead).

nex3 commented 6 years ago

I don't have all of this loaded into my head anymore, but I believe the general idea is to have pub auth <host> authenticate with Google on behalf of the given host, so that each host has its own set of Google credentials and none of them can spoof the other hosts. There are a lot of details beyond that that would need to be figured out, though.

raveesh-me commented 5 years ago

dependencies:
  kittens:
    git:
      url: git://github.com/munificent/kittens.git
      ref: some-branch

This is how we add a dependency from a github/gitlab repo How we do it when the repo is private? Based on the thread this is possible but can we have updated doccumentation for the same?

https://www.dartlang.org/tools/pub/dependencies#git-packages Does not help. Can someone please guide me on how to achieve this?

CC: @kevmoo

travissanderson-wf commented 5 years ago

@raveesh-me this issue is actually about hosting a 3rd party pub server (like pub.dartlang.org). However, if you use the ssh style URL instead it should work fine (assuming your user has access to the repo):

dependencies:
  kittens:
    git:
      url: git@github.com:munificent/kittens.git
      ref: some-branch

raveesh-me commented 5 years ago

@travissanderson-wf Thanks a lot! I actually figured that out thanks to gitter.

I realise the purpose of this issue. Sorry for going off topic on this thread. Filing a new bug for updating the docs.

thosakwe commented 5 years ago

Not 100% sure of the status of this issue, but in theory, the ~/.pub-cache/credentials.json file could be changed to look like this format, and could host credentials for different hosts in the same file:

{
  "pub.dartlang.org": {
    "accessToken": "..."
  }
}

I guess the client could auto-convert existing files by checking for an accessToken key, and then re-writing the credentials.json to allow users to remain signed in.

jonasfj commented 5 years ago

@thosakwe, yeah, that would be a decent approach... however, it still leaves open the question of how users authenticate with a server other than pub.dartlang.org.

I suppose it might be acceptable to simply prompt the user for an "authToken", store said token in the $PUB_CACHE/credentials.json file and attach it as authorization: bearer <authToken> when making requests. In this scenarios I'm suspecting that "authToken" would be an opaque string the user obtains from the third-party package repository.

So for using pub.dartlang.org we would still have a hardcoded flow, but for any other package repository we would require that users open a browser, sign into the package repository's website, and create an "authToken". Similar to how some websites will let you create an apiKey/apiToken/authToken... A possible downside would be a missing refresh flow, so tokens would have to not expire or have a lengthy expiration.

thosakwe commented 5 years ago

Yeah, I think that probably just using an auth token would work just fine for any host other than pub.dartlang.org.

As for refresh tokens, two ideas popped into my head:

Potentially prompt the user for one if the current token has expired
More complicated, probably not so practical: 3rd-party servers could describe how Pub should interact with them, maybe at a /.well-known/pub.json URI:

{
  "oauth2": {
    "authorizationUrl": "https://pub.example.com/oauth2/authorize",
    "tokenUrl": "https://pub.example.com/oauth2/token",
     // Potentially even provide different credentials for Pub to use (optional)
     "clientId": "...",
     "clientSecret": "..."
  }
}

Of course, there is also the option of just not supporting refresh tokens.

jonasfj commented 5 years ago

Yeah, having an API end-point fetching the client secret, in combination with /.well-known/openid-configuration is tempting... I haven't studied the spec in detail, but I think a client secret is still needed at least for Google.. maybe it's possible to require that servers implement openid-connect without requiring a client secret? And maybe the Google flow could work by having our server redirect to that, so only it knows the client secret... (These ideas are not fully thought through)

In any event, I worried we're over-engineering this..if we could do the command interface such that we can add support for an automated login flow later that might be nice.

noahbetzen-wk commented 5 years ago

I made https://github.com/dart-lang/pub_server/issues/39 not realizing that this issue had been made. This would be a super useful feature.

thosakwe commented 5 years ago

FWIW, I didn’t realize it before, but it’s possible to include credentials in the PUB_HOSTED_URL, and use that for Basic auth, etc.

https://pub.dartlang.org/packages/mpp

The downside, of course, is including the credentials in an environment variable.

I think I might get started on a PR for patching the credentials.json format...

jonasfj commented 5 years ago

I've discussed this IRL with a few people and consensus is that:

pub should use an opaque token for authentication with 3rd party servers.
The opaque token is attached as authorization: bearer <token> in authenticated requests.
An opaque token is stored for each hostname in $PUB_CACHE/tokens.json (other than pub.dartlang.org and pub.dev, which will use credentials.json).
pub should prompt users for this token on the command line interface, if not present in $PUB_CACHE/tokens.json.

This means that pub.dartlang.org and pub.dev will have special treatment, where as for all other pub servers the flow will be as follows:

User starts command that requires authentication (e.g. pub publish).
If $PUB_CACHE/tokens.json doesn't contain a token for the pub server (identified by hostname), then:
- Prompt user for a token (asking the user to obtain this from the server out of band).
- Associated token with hostname for server in $PUB_CACHE/tokens.json.
Use token associated with hostname for server from $PUB_CACHE/tokens.json.
Send request associating token using authorization: bearer <token>.

@thosakwe, if you're interested in working on this let me know, I'll be happy to answer questions around the design, and help with reviews. This is definitely a contributable features and I think we want it :)

sigurdm commented 5 years ago

Consider: attempting without a token first, and when receiving a 401 ask the user for a token, with a message from the server's reply. This allows a custom server to guide the user to obtain a token.

thosakwe commented 5 years ago

Rolling in this branch: https://github.com/thosakwe/pub-1/tree/external-auth Related pull request (currently a draft): https://github.com/dart-lang/pub/pull/2167

themisir commented 3 years ago

Why this is taking so long? Is there any ETA or something planned for this issue?

Just wondering if there's any issue that' blocking this one. If not I'm planning to implement it.

tusharojha commented 3 years ago

Though @jonasfj knows it better, I guess it will be solved by this summer.

jonasfj commented 3 years ago

knows it better, I guess it will be solved by this summer.

We know today that this will not be part of GSoC 2021.

If someone wants to work on it anyways, you are welcome.. maybe drop me a line in hackers-pub-dev-

themisir commented 3 years ago

I've prepared a draft proposal doc for hosted pub authentication. Looking forward for your feedbacks.

https://github.com/TheMisir/pub/blob/master/doc/authentication-proposal.md

dart-lang / pub

Support authenticating pub against 3rd-party servers #1381