cmd/go: define HTTP authentication extension mechanism

draftcode commented 6 years ago

NOTE: The accepted proposal is https://github.com/golang/go/issues/26232#issuecomment-461525141.

Problem

The custom import path mechanism (?go-get=1 and meta tag) works for public URLs, but it doesn't work for auth required URLs. This is because when go-get fetches the URL with ?go-get=1, it uses net/http.DefaultClient, and it doesn't know the credentials it needs to access the URL. A user cannot run go get against private source code hosting service because of this.

Goal

Make go get git.mycompany.com/private-repo work, where https://git.mycompany.com/private-repo requires authentication.

Idea 1 (credential helper)

Introduce a credential helper mechanism like git-credential-helpers. A user specifies a path to a binary via GOGET_CREDENTIAL_HELPER and go get executes that with the import path as an argument. The credential helper binary returns HTTP headers (like "Authorization: Basic blah\n", and go get adds these headers when it tries to fetch go-get=1 URLs.

PRO: Straightforward solution to the problem description.
CON: Supporting multiple credential helpers becomes complicated. Git's credential helper mechanism supports multiple credential helper, and Git runs each in order. This sometimes makes an unexpected behavior that is hard to debug.

Idea 2 (go-get helper)

Introduce a custom source code fetching mechanism. When go get needs to fetch the source code for the import path git.mycompany.com/private-repo, it looks for the binary go-get-git.mycompany.com based on the host name of the import path. When such binary exists in $PATH, it executes that binary with the import path and a destination in $GOPATH (for example, go-get-git.mycompany.com git.mycompany.com/private-repo $GOPATH/src/git.mycompany.com/private-repo). The binary is responsible for fetching the source code to the specified $GOPATH location.

PRO: As a side effect, this make go get work with VCSs other than git/hg/svn/bzr.
CON: I'm not sure how this works with go get -f or go get -insecure.

gopherbot commented 5 years ago

Change https://golang.org/cl/170879 mentions this issue: cmd/go/internal/web{,2}: consolidate web packages

gopherbot commented 5 years ago

Change https://golang.org/cl/170880 mentions this issue: vcs-test/vcweb: fix various extraction issues for serving modules with authentication

rsc commented 5 years ago

Looks good. On Feb 13 I asked if anyone objected to this proposal, and as far as I can tell no one has. Accepting.

palsivertsen commented 5 years ago

You only get to pick two of those: “requires authentication” and “authenticating […] is not an option” are mutually exclusive.

Sorry, poor choice of words from my side. What I was trying to say is that HTTP is not an option, but other protocols are, e.g. SSH. Making HTTP calls that requires authentication is (IMO) a poor choice compared to SSH because it requires either user interaction (2FA) or weak credentials (compared to SSH). Another argument to drop HTTP is that SSH credentials are in most cases only authorized for GIT, but HTTP credentials in some cases gives access to a wider API.

Maybe my usecase is not relevant to what's described in this issue.

bcmills commented 5 years ago

@palsivertsen, the protocol-selection issue you describe was discussed in #30304 (and declined on the grounds of not being worth the extra complexity; see https://github.com/golang/go/issues/30304#issuecomment-470597610).

gopherbot commented 5 years ago

Change https://golang.org/cl/172599 mentions this issue: cmd/go/internal/base: add a Logf function and funnel stderr output through it

gopherbot commented 5 years ago

Change https://golang.org/cl/172617 mentions this issue: cmd/go: set GIT_TERMINAL_PROMPT on individual git commands instead of process-wide

hyangah commented 5 years ago

@bcmills I want to make sure I understand how this protocol works with GOPROXY and private repos. Correct me if wrong.

Option 1. Utilizing the functionalities added for #26334. User specifies GOPROXY=https://public_proxy,direct

Go command tries to fetch https://public_proxy/private_repo/package/path
The public_proxy will not be able to access the private repo, so returns 4XX.
Go command runs $GOAUTH with https://public_proxy/private_repo/package/path as an arg, which fails.
Go command tries with the next one in GOPROXY (direct in this example).
The private repo requires credential, so returns 4XX.
Go command runs $GOAUTH with https://private_repo/package/path as an arg, which succeeds.
Go command fetches data from the private repo using the credential.

Option 2. Utilizing the new GONOPROXY env var proposed in https://go.googlesource.com/proposal/+/master/design/25530-notary.md#command-client. User specifies GONOPROXY=private_repo GOPROXY=https://public_proxy

Go command knows the private_repo shouldn't use the proxy, so directly contact the private_repo, and fails with 4XX (403)
Go command runs $GOAUTH with https://private_repo/package/path and acquires the credential.
Go command fetches data from the private repo using the credential.

bcmills commented 5 years ago

@hyangah, since GOAUTH may be interactive, I think we want to avoid running it until we have exhausted all other options. For your Option 1, that suggests something more like:

Go command tries to fetch https://public_proxy/private_repo/package/path
The public_proxy will not be able to access the private repo, so returns 4XX.
Go command tries with the next one in GOPROXY (direct in this example).
The private repo requires credential, so returns 4XX.
Go command runs $GOAUTH with https://public_proxy/private_repo/package/path as an arg, which fails, so it does not retry the public proxy.
Go command runs $GOAUTH with https://private_repo/package/path as an arg, which succeeds.
Go command fetches data from the private repo using the credential.

That reduces the number of user interactions required if the proxy declines the request but the direct fetch succees. However, it implies that if one wants to avoid leaking paths from one proxy to another, the chosen GOPROXY command must report all applicable credentials upfront (during the initial “no URL” phase) instead of waiting for requests for specific URLs.

hyangah commented 5 years ago

Thanks for the clarification, @bcmills

However, it implies that if one wants to avoid leaking paths from one proxy to another, the chosen GOPROXY command must report all applicable credentials upfront (during the initial “no URL” phase) instead of waiting for requests for specific URLs.

Does that mean, if the GOAUTH program returns the credential for https://private_repo/ during the initial no URL phase, the Go command will not try to use GOPROXY for the package that prefixes with the private_repo so the private package path and repo names will not leak? That's great.

If I have some private repositories in github.com accessible with my github.com credential, but I still want to utilize the public GOPROXY for other public modules and packages, do I need to configure the GOAUTH program to return the explicit list of private repositories in the github.com with my github.com credentials?

gopherbot commented 5 years ago

Change https://golang.org/cl/173017 mentions this issue: cmd/go: query modules in parallel

bcmills commented 5 years ago

Does that mean, if the GOAUTH program returns the credential for https://private_repo/ during the initial no URL phase, the Go command will not try to use GOPROXY for the package that prefixes with the private_repo so the private package path and repo names will not leak?

I'm not sure exactly what you're asking, but I wasn't planning to use the set of credentials to decide which modules to ask the proxy about.

If I have some private repositories in github.com accessible with my github.com credential, but I still want to utilize the public GOPROXY for other public modules and packages, do I need to configure the GOAUTH program to return the explicit list of private repositories in the github.com with my github.com credentials?

If you don't care about leaking your private repo paths to the proxy, then you can run git config --global credential.helper cache or git config --global credential.helper 'store --file ~/.my-credentials', then git credential approve the credentials for your private repos, and finally set GOAUTH=git to use those stored credentials. (To my knowledge git does not have a API for enumerating existing credentials, so the GOAUTH initial pass won't pick them up anyway.)

If you do care about leaking your private repo paths, then you'll still have to use GONOPROXY or similar.

bcmills commented 5 years ago

I had to pause development on this before the 1.13 freeze due to a personal emergency, and as a result it didn't make the freeze.

Given that it appears likely that modules will still be auto rather than on by default (#31857), and given the subtle interactions between GOAUTH, package-to-module resolution, and proxy fallback (#26334), I think this needs to wait for 1.14.

bcmills commented 4 years ago

Just to set expectations: I don't think this will make the 1.14 cycle. (There is too much left to do, and the fixed .netrc support in 1.13 seems to have addressed many of the popular hosting platforms.)

urandom2 commented 4 years ago

Would it be possible to break up work so that the community can contribute? I am interested in this issue, but unclear if I could meaningfuly help, short of just landing the feature in isolation.

bcmills commented 4 years ago

@arnottcr, I don't know of any way to break it up. The bulk of the work, I think, is making sure that everything works properly in conjunction with the GOPROXY fallback sequence and the package-to-module search sequence.

(Essentially, we currently have a two-dimensional search space, and GOAUTH adds a third dimension to search, so we need to figure out exactly what order to traverse those dimensions to avoid breaking things like the GOPROXY privacy properties.)

nkralles commented 4 years ago

My teams biggest blocker to migrate to use go modules internally right now is not being able to define a client certificate for authentication. We have all of repositories behind a proxy that is set to SSLVerifyClient:required. All of our git clients are set up to provide client certificates or use just plain ssh. Could the go get when setting GOAUTH=git respect the http configurations defined in the .gitconfig for ssl client authentication be passed on to the go-get=1 request?

.gitconfig

[http]
    sslCert = <path to cert>
    sslKey = <path to key>
    sslVerify = true/false
    sslCAInfo = <path to ca>

bcmills commented 4 years ago

@nkralles, this issue is not about client certs. (That is #30119.)

firelizzard18 commented 4 years ago

Do you have an estimate for when this will be implemented? E.g. 1.15 or 1.16?

ericsampson commented 3 years ago

Are there any more general "HTTP auth credential helpers" in the world besides git-credential-helper? How do other systems deal with this? If there's something standard to hook into (like .netrc) then that's better than designing our own.

I'm way late to this party, but in case it's of any interest, this is pretty much what Microsoft's NuGet package managers (nuget.exe and dotnet CLI and Visual Studio) do for authenticated package feeds using a shared nuget credential provider

mjrlee commented 3 years ago

I'm butting up against this issue in a slightly different way.

We proxy all our traffic to our git server through a kerberos authenticating proxy. Most applications use libcurl at some point, which handles this nicely (performs negotation after receiving a 401).

https://github.com/jcmturner/gokrb5 has a pure go implementation of SPNEGO. I'd be up for writing some GOAUTH handler that generates a preemptive negotiate header.

shakefu commented 2 years ago

Do you have an estimate for when this will be implemented? E.g. 1.15 or 1.16?

Go 2? ... Would really be nice to see Go and its toolchain have the same first class support for standard auth mechanisms.

srowles commented 2 years ago

Just bumped into this while trying to get at a gitlab instance that was behind a cloudflare auth. So commenting here to add to the use-cases.

I'd like to be able to set, somehow, a custom http header to allow cloudflare access e.g.

cf-access-token: $TOKEN

If I could configure this custom header, in a .netrc style, so that I can just say that matching hosts have to supply the extra header value, it would have solved the access problem early nicely

pboguslawski commented 2 years ago

Client TLS cert auth to private git repos would be nice also: https://github.com/golang/go/issues/53197

Dansyuqri commented 2 years ago

Seems like there are a few ways to do authentication, one of them using the header to send a token. I am also facing the same issue as @srowles . As such, would it be useful to split this task into multiple smaller ones? One of them to add the reading of a header and its corresponding value from Go's environment variable (i.e. GOAUTH_HEADER, GOAUTH_HEADER_VALUE)?

owenhaynes commented 2 years ago

Looks like GCP are trying to do stuff with artefact registry now and go module proxy. https://github.com/GoogleCloudPlatform/artifact-registry-go-tools

So looks like there is a need to support something

AndrewBurian commented 1 year ago

I am very late to this party it seems, but I might have a bit more to offer than just a +1.

netrc as the only supported authentication mechanism for module proxies doesn't seem to be a passionate issue anywhere, but like @owenhaynes pointed out, module adoption is widespread and having to manage git/ssh credentials for private modules is a growing thorn. I'll go as far as to say I think programatically updating .netrc is probably more a sign of desperate madness than a robust solution.

For better or worse, I didn't find this issue until after I'd taken a stab at fixing it myself. Fortunately, I came to most of the same conclusions this proposal did, with a few exceptions. I agree entirely with avoiding bespoke protocols where good ones exist, but also that existing protocols that leverage plaintext and hardcoded tokens are best left behind.

The few assumptions I think I might have made differently to @bcmills:

it is safe to assume prior knowledge of which helpers will be able to provide credentials for what proxies
we avoid interactiveness through caching, but if a proxy has a defined credential helper, we don't even try before fetching a cred
the myriad of ways that authentication can happen is not a problem space that can be captured in advance, and the protocol for pulling "credentials" needs to be gracefully extensible so future versions of Go can support new mechanisms (e.g. mTLS) without breaking changes to the interaction between go and the helpers

My design is as follows:

The GOAUTH config behaves like GOVCS in that it's an associative config that maps proxy hosts to helpers that can supply them credentials.

The grammar for GOAUTH is GOAUTH=<proxy-glob>!<helper-bin>[;<arg>[; ...]][, ...]. By example:

GOAUTH='*priv.example.com!/usr/local/bin/corp-credhelper;--user=me,proxy.example.com/private/*!my-credtool'

In this, proxy-glob is a host and path that will be matched with module.MatchPrefixPatterns, and an associated helper-bin that is executed with any corresponding arguments. Earlier matches take precedence, and go only invokes one helper per request.

The protocol between go and the helper is borrowed from a system that seems to be working well in the real world: kubernetes and its client.authentication.k8s.io ExecCredential. This is the helper system that kubectl commands use to authenticate to clusters.

GOAUTH plugins must return a HttpCredential object as JSON over stdout. Go will ignore any unknown fields in this object, allowing for future extensibility without breaking changes. As I've defined it, it supports the arbitrary http headers approach, a shorthand for Bearer tokens, and basic auth. I was going to suggest TLS client cert/secret pairs and CA bundle would be a worthy addition as https://github.com/golang/go/issues/26232#issuecomment-461193285 mentioned.

The HttpCredential response includes information about how long the provided credentials will be valid for, allowing Go to cache it both in-memory for repeat requests, or on-disk for caching between commands. GOAUTHCACHE points to a file on disk (default to $USER_CONFIG_DIR/.goauthcache) where it's stored. Since I'm not a big fan of writing creds to disk if I don't have to, I am assuming most cred helpers can either choose to handle the caching themselves, or the disk cache can be disabled with GOAUTHCACHE=off.

Test code

I have this implementation "working" in a dev branch: https://github.com/golang/go/compare/master...AndrewBurian:go:auth-plugins

This is my first time seriously slogging my way through go's test suite, so forgive... everything. It's mostly a PoC so I could see if the design was even a good idea or not, and seems to work as I'd expect.

Git+SSH?

I'm not sure if this helps or hurts the situation with Github and ssh auth #26134.

My understanding after catching up all the discussion here is that go defers everything about the transport of source code outside of module zips to the VSC tool, and so having to configure git separately with auth for modules being served from private VCS feels correct. The focus of GOAUTH is very much for the distribution of private modules, and GOPRIVATE for private VCS.

There may be some weird behaviour if the http authentication allows go to discover the existence of a repo with the initial https go-get=1 request, but then not be able to clone it if git hasn't had credentials configured. I got turned around in the code, but it seems like with the hardcoded VCS handler for Github, go maybe doesn't even bother with the initial get?

I will continue to play around with this, but after finally getting it to all compile my go seamlessly fetches modules with GOPROXY set to only a private Google Cloud Storage bucket and a 3-line script wrapping up gcloud auth print-access-token as a cred helper. In my opinion, it feels like how I would expect private authentication to work coming from git and k8s.

seankhliao commented 1 year ago

maybe instead of implementing the myriad of authentication strategies people may want, we just have a making of module prefixes to plain http proxies and let the proxies be responsible for the auth

nkralles commented 1 year ago

The lack of the ability to set client certificates for the module resolver to connect to modules behind mTLS repos/gitlab/etc is the reason I am currently forced to use a proxy. Being able to set client side certificates and certificates authorities would be a nice addition and would eliminate the need for those like me who are forced to proxy around the security that mTLS provides.

On Wed, Mar 22, 2023 at 11:31 AM Sean Liao @.***> wrote:

maybe instead of implementing the myriad of authentication strategies people may want, we just have a making of module prefixes to plain http proxies and let the proxies be responsible for the auth

— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/26232#issuecomment-1479785279, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPPCRM6N4ZR4FAE3NIODQ3W5MLNVANCNFSM4FIQKQQQ . You are receiving this because you were mentioned.Message ID: @.***>

rsj-ioki commented 1 year ago

The GOAUTH env var is not yet recognized right? At least on 1.20.4 it is being ignored

alorle commented 1 year ago

Any news on this?

I've seen that this has been worked on (https://github.com/golang/tools/tree/master/cmd/auth), but as @rsj-ioki says it does not seem that it is fully implemented or available.

silverwind commented 1 year ago

The lack of any authentification mechanism against a GOPROXY is a pretty serious security concern for anyone running their own private proxy because anyone with HTTPS access to the proxy can retrieve all stored modules.

mxk commented 4 months ago

Whoever works on this, please make sure that pkgsite commands use the same mechanism. Currently, it is very difficult to get a private documentation server running, and even once it's up, the lack of even netrc support causes all source links to be broken.

gopherbot commented 1 month ago

Change https://go.dev/cl/605256 mentions this issue: cmd/go: add GOAUTH mechanism for HTTP authentication

gopherbot commented 1 month ago

Change https://go.dev/cl/605275 mentions this issue: cmd/go: add built in git mode for GOAUTH

gopherbot commented 1 month ago

Change https://go.dev/cl/605298 mentions this issue: cmd/go: add user provided auth mode for GOAUTH

golang / go