Closed kevmoo closed 3 years ago
Any idea what authentication schemes would be supported?
I'm moving speakeasy into its own org, https://github.com/speakeasy-pub/speakeasy, and would be very interested in this conversation.
I've had success at work with https://github.com/rlidwka/sinopia for NPM and that has multiple authentication backends. Open to doing something similar with speakeasy.
My understanding is that they would extend the same OAuth2 functionality that already exists for the pub publish
command.
That's right. The plan is to add a pub authenticate HOSTNAME
command that will send the user through the OAuth2 authentication process, eventually providing a token that the given host can use to verify the user's google account. This token will be stored in the user's pub cache, and sent along with any requests to that server in the course of version resolution.
For small organizations like us, it's really beneficial to use Google's identity management. And I'll be happy with it. But just of out curiosity and for the larger organizations than us requiring use of their own id management, does it require much effort to implement support for oauth2 id service other than Google's?
There are some technical constraints that make it difficult. OAuth2 requires that the client—in this case pub—have a client ID and secret provided to it out-of-band by the identity provider. We already have such an ID and secret for Google built in to pub, but getting it out-of-band for arbitrary identity servers is harder. Not impossible, but not necessary for what we're doing here either.
Maybe a first step could be to enable authentication for pub get using the same authentication mechanism that is already used for pub publish? E.g.
export PUB_GET_AUTHENTICATION_ENABLED=true
pub get
# will use ~/.pub-cache/credentials.json just like pub publish
We run our own dart pub package server and we are fine with using google oauth for pub publish. Our server verifies the authorization token against Google. The returned user is checked against a list of allowed uploaders. Adding pub get authentication would give us what we need to protect the repository from unauthorized users.
We can't send users' pub.dartlang.org auth tokens to other servers; it would allow any server that they download packages from to spoof them and upload packages on their behalf.
What if pub get tokens could be restricted to PUB_HOSTED_URL so it is only on intent that it is used?
Similar to how pub publish behaves right now. It is sending the pub.dartlang.org auto tokens to our server since we override it:
export PUB_HOSTED_URL=https://...
That also opens up for spoofing in a sense. Not that it matters to us because we are completely aware of this and trust our own pub package server.
I'm not comfortable assuming that all users will be completely aware of what they're doing in that case. The only way I'm willing to do this is if each hosted URL has its own credentials.
Ok, makes sense.
any movement on this @nex3 @kevmoo ?
Not yet. I'm hoping to budget time for it next quarter.
Any update on this?
We still want to do this – but we likely won't get to it until early 2017.
I'm willing to help out with this.
We'd love help here – but it's critical we have a design that's approved upfront.
If you're interested, @thosakwe ...
I have a fork of pub called "re:pub" that supports basic auth, in addition to Google OAuth2.
Perhaps users could store their basic auth credentials in a file, which could be read as follows:
# .pub-auth.yaml
username: jdoe1
password: foobarbaz24
And then, run pub with an argument to point to the file. This is assuming the user has set a "PUB_HOSTED_URL."
pub publish --auth-file .pub-auth.yaml
And then, perhaps users could optionally specify a host in the YAML file, so they wouldn't have to set an environment variable every time.
This design would make it possible to use dependencies from public pub, as well as publishing to a private host.
However, there would need to be some sort of distinction made between private and normal dependencies. I think that in the "pubspec", private dependencies would have an extra attribute added, whereas normal ones would remain the same.
# pubspec.yaml
dependencies:
string_scanner: ^1.0.0 # normal
in_house_package:
hostname: https://pub.my-private-company.com
version: ^0.3.5
Let me know what you think!
I'm not a big fan of adding a bunch of extra configuration files. We already have .pub-cache/credentials.json
as a global config file, so I think any new configuration should go in there. I'd like the user's interaction with it to all go through the pub
executable—for example, pub auth add http://example.org
could authenticate the user with that domain.
We already have means of depending on hosted packages from arbitrary domains, and of specifying what domain a package should be published to. I don't think we need to add extra functionality for that.
Currently speakeasy does caching of packages from pub.dartlang.org.
NPM has scoping which lets you easily target multiple registries if you need to. Not sure how pub would do that at the moment.
I'm not sure how to handle scoping, either but...
In response to @nex3: On second thought, adding additional credentials to the .pub-cache/credentials.json
is better all-around.
I feel that it might be easier to support Basic Auth for third-party hosts. Otherwise, every third-party host would have to implement the OAuth2 spec, and Pub would have to remember a new Client ID and Client Secret for each host.
Perhaps Pub could support an HTTP header, X-Pub-Auth-Required
, or something similar. If the server returns a 403
on a request, and X-Pub-Auth-Required
is set to basic
, then Pub would prompt the user for a username and password, and retry the request. If another 403
is thrown, then Pub would abort the request. In this case, the server could send another HTTP header to describe the authentication error, X-Pub-Auth-Error
: Incorrect password.
Otherwise, a default error message could be shown.
Just a thought. I do think this would work, though.
I have usability concerns about supporting basic auth. Either we'd have to make users authenticate for every invocation of pub
, which is a very substantial usability burden; or we'd need to store the user's raw credentials in credentials.json
, which isn't a good security practice and may not be clear to users, since we don't store pub.dartlang.org credentials that way. One of the major benefits of a system like OAuth2 is that the stored information doesn't make the user's password visible, and I'd be unhappy to give that up.
Ah, I understand. So in that case, would it be difficult to store a client ID/secret, as well as an access token for each third-party host?
Is any work currently being done on this? Could someone recommend a solution for private packaging?
Currently the best solution is to use private Git repositories.
Private git repositories work pretty well, the primary shortcoming is that pub can't do version range resolution with those. Until auth is added, we are running our private pub server with local network access only. We have a huge selection of private libraries so range resolution is a must at this point.
I have a private repo for my components. Is it possible to use pub to retrieve it though? If not, how would I references a directory or repo?
dependencies:
my_package:
git: git://url
You can also use an HTTPS URL there, i.e. Github
Note that this currently only works for one package per repo. I think there is an improvement coming in the next SDK to let you define a sub-directory.
Here's the docs for the current method of declaring git dependencies. This is the improvement coming in Dart 1.25 to allow depending on specific directories in a git repo:
Git dependencies may now include a path parameter, indicating that the package exists in a subdirectory of the Git repository
@travissanderson-wf @thosakwe Thanks guys, I trolled through a lot of documentation, but I didn't see this. I appreciate you steering me in the right direction.
This issue still causes Workiva pain on a regular basis. Without OAuth2 support, we're forced to whitelist access to our pub server by IP address. This involves many pain points:
pub get
. This is the only part of our entire development environment (including python, go, and java package repositories) that requires developers to VPN in.Would be happy to discuss this issue further if needed! :)
This is still something I think would be beneficial, but we haven't had any resources to allocate towards it. If someone externally wanted to work on it I'd be happy to review.
@nex3: do you have some pointers on what the scope and the extent of such work could be? A friend have complained to me about the inability of using a private pub server with configurable auth (which contributed to their decision to go ahead with the npm world instead).
I don't have all of this loaded into my head anymore, but I believe the general idea is to have pub auth <host>
authenticate with Google on behalf of the given host, so that each host has its own set of Google credentials and none of them can spoof the other hosts. There are a lot of details beyond that that would need to be figured out, though.
dependencies:
kittens:
git:
url: git://github.com/munificent/kittens.git
ref: some-branch
This is how we add a dependency from a github/gitlab repo How we do it when the repo is private? Based on the thread this is possible but can we have updated doccumentation for the same?
https://www.dartlang.org/tools/pub/dependencies#git-packages Does not help. Can someone please guide me on how to achieve this?
CC: @kevmoo
@raveesh-me this issue is actually about hosting a 3rd party pub server (like pub.dartlang.org). However, if you use the ssh style URL instead it should work fine (assuming your user has access to the repo):
dependencies:
kittens:
git:
url: git@github.com:munificent/kittens.git
ref: some-branch
@travissanderson-wf Thanks a lot! I actually figured that out thanks to gitter.
I realise the purpose of this issue. Sorry for going off topic on this thread. Filing a new bug for updating the docs.
Not 100% sure of the status of this issue, but in theory, the ~/.pub-cache/credentials.json
file could be changed to look like this format, and could host credentials for different hosts in the same file:
{
"pub.dartlang.org": {
"accessToken": "..."
}
}
I guess the client could auto-convert existing files by checking for an accessToken
key, and then
re-writing the credentials.json
to allow users to remain signed in.
@thosakwe, yeah, that would be a decent approach... however, it still leaves open the question of how users authenticate with a server other than pub.dartlang.org
.
I suppose it might be acceptable to simply prompt the user for an "authToken", store said token in the $PUB_CACHE/credentials.json
file and attach it as authorization: bearer <authToken>
when making requests. In this scenarios I'm suspecting that "authToken" would be an opaque string the user obtains from the third-party package repository.
So for using pub.dartlang.org
we would still have a hardcoded flow, but for any other package repository we would require that users open a browser, sign into the package repository's website, and create an "authToken". Similar to how some websites will let you create an apiKey
/apiToken
/authToken
... A possible downside would be a missing refresh flow, so tokens would have to not expire or have a lengthy expiration.
Yeah, I think that probably just using an auth token would work just fine for any host other than pub.dartlang.org
.
As for refresh tokens, two ideas popped into my head:
/.well-known/pub.json
URI:{
"oauth2": {
"authorizationUrl": "https://pub.example.com/oauth2/authorize",
"tokenUrl": "https://pub.example.com/oauth2/token",
// Potentially even provide different credentials for Pub to use (optional)
"clientId": "...",
"clientSecret": "..."
}
}
Of course, there is also the option of just not supporting refresh tokens.
Yeah, having an API end-point fetching the client secret, in combination with /.well-known/openid-configuration is tempting... I haven't studied the spec in detail, but I think a client secret is still needed at least for Google.. maybe it's possible to require that servers implement openid-connect without requiring a client secret? And maybe the Google flow could work by having our server redirect to that, so only it knows the client secret... (These ideas are not fully thought through)
In any event, I worried we're over-engineering this..if we could do the command interface such that we can add support for an automated login flow later that might be nice.
I made https://github.com/dart-lang/pub_server/issues/39 not realizing that this issue had been made. This would be a super useful feature.
FWIW, I didn’t realize it before, but it’s possible to include credentials in the PUB_HOSTED_URL, and use that for Basic auth, etc.
https://pub.dartlang.org/packages/mpp
The downside, of course, is including the credentials in an environment variable.
I think I might get started on a PR for patching the credentials.json
format...
I've discussed this IRL with a few people and consensus is that:
pub
should use an opaque token for authentication with 3rd party servers.authorization: bearer <token>
in authenticated requests.$PUB_CACHE/tokens.json
(other than pub.dartlang.org
and pub.dev
, which will use credentials.json
).pub
should prompt users for this token on the command line interface, if not present in $PUB_CACHE/tokens.json
.This means that pub.dartlang.org
and pub.dev
will have special treatment, where as for all other pub servers the flow will be as follows:
pub publish
).$PUB_CACHE/tokens.json
doesn't contain a token for the pub server (identified by hostname), then:
$PUB_CACHE/tokens.json
.$PUB_CACHE/tokens.json
.authorization: bearer <token>
.@thosakwe, if you're interested in working on this let me know, I'll be happy to answer questions around the design, and help with reviews. This is definitely a contributable features and I think we want it :)
Consider: attempting without a token first, and when receiving a 401 ask the user for a token, with a message from the server's reply. This allows a custom server to guide the user to obtain a token.
Rolling in this branch: https://github.com/thosakwe/pub-1/tree/external-auth Related pull request (currently a draft): https://github.com/dart-lang/pub/pull/2167
Why this is taking so long? Is there any ETA or something planned for this issue?
Just wondering if there's any issue that' blocking this one. If not I'm planning to implement it.
Though @jonasfj knows it better, I guess it will be solved by this summer.
knows it better, I guess it will be solved by this summer.
We know today that this will not be part of GSoC 2021.
If someone wants to work on it anyways, you are welcome.. maybe drop me a line in hackers-pub-dev-
I've prepared a draft proposal doc for hosted pub authentication. Looking forward for your feedbacks.
https://github.com/TheMisir/pub/blob/master/doc/authentication-proposal.md
We've discussed something like
pub authenticate example.com
This would allow private pub repos to have authenticated access.
We'd also want authentication to be supported for
pub get/update
CC @computmaxer