Building plugins from multiple private repos

[yroc92]

I have a use-case where I want to build Caddy with plugins from multiple disparate private repos (different accounts / organizations). As an example of why this might be useful, consider the scenario when xcaddy is used in an automated build environment where users can link their private repo via Oauth to create a custom Caddy build.

Retrieving a plugin from a single private repo is trivial (just set GOPRIVATE and do a replace on the git remote url host, or use SSH). Multiple private plugins from different organizations would require you to make the the aforementioned changes to the environment just before each go get call.

I'm not sure the best way to solve this. My idea is to have xcaddy accept a JSON file as a param. It could include an array of "private repo configs" which each consist of the following

• User name
• Password / token
• Repo url

xCaddy would use this info to

1. Set GOPRIVATE env var (used by go get to know when a repo is private),
2. Assemble a temporary .netrc before each go get call for a private plugin (the .netrc is used by the git command to authenticate with the git server.

The json config could be nice because would be simple to manage in a CI/CD environment or in your Dockerfile.

Caveat: Right now I'm only thinking about private Github repos, so there might be some minor nuances to supporting / authenticating with other Git servers.

Here's an article about go get and private repos for more context.

[francislavoie]

Could we do a weird syntax like --with github.com/org/repo=!<user>:<pass>@<repo>? Basically ! as an early flag for the replacement thing to instead trigger private repo "mode" for the go get 🤔 the special char used for that is up in the air

[yroc92]

Github at least has an official syntax that does what you're describing

https://<token>@github.com/owner/repo.git

I wonder if we should just have xcaddy support that? I tried running xcaddy with that and it doesn't work:

malformed module path "<redacted token>": missing dot in first path element

[francislavoie]

Can you try with go env -w GOPRIVATE=github.com/org/repo? Will it work if you just do that? That should override your env to set that for all subsequent Go commands, which should make xcaddy use it as well.

[yroc92]

It does, but I'd still need to authenticate with my token.

[francislavoie]

Right, then I guess that's what the .netrc is for.

[yroc92]

Right, but imagine you need to authenticate to several private repos because your build requires several private plugins. Then the .netrc would need to change before each go get call that caddy makes as it iterates through the requested plugins.

[francislavoie]

You can have multiple accounts in a netrc file, can't you? I've never used it before but a quick Google suggests you should be able to do that.

https://stackoverflow.com/a/50771012

[francislavoie]

So I assume that did the trick?

[yroc92]

I got sidetracked but this recently became relevant again. The problem with the .netrc file is that it only allows for 1 credential. Imagine you want to build Caddy with 2 private plugin repos, each from a different account (perhaps you're a PaaS building Caddy with your own private plugins and customer plugins). With xcaddy, the only workaround would be to use the =<replacement> feature and clone the repos locally with the oauth repo syntax (github lets you use a "personal access token" that you can generate in your settings). You can see an example here.

In addition to the .netrc file and using the oauth syntax to clone repos and pointing xcaddy to the local copy, you can also authenticate via ssh. You can also combine these methods. For example, this Dockerfile is writing a .netrc with some provided credentials and then running the example script above with an oauth token to clone repos from a different account.

The problem with the .netrc and ssh keys is that you can't have multiple keys for a single host (github.com). How do we scale this without writing / replacing stuff in the filesystem? The safest way, in my opinion, is to clone everything locally and use replacements in the --with flags, as seen in my example above.

Here's an idea of an interface to solve this:

A --credentials flag that accepts JSON / JSON file like so:

[
  {
    "credential": "<personal access token for account A>",
    "repos": ["github.com/private-repo-a/someplugin1", "github.com/private-repo-a/someplugin2"]
  },
  {
    "credential": "<personal access token for account B>",
    "repos": ["github.com/private-repo-b/someplugin3", "github.com/private-repo-b/someplugin4"]
  }
]

and behind the scenes it would be cloning and using replacements. I don't love this, but after researching this issue for many hours, there's not really anything that scales well 🤷🏿 .

We could also go the direction of a more generic --plugins flag that accepts json for all the plugins & optional credentials which would replace the need for writing many --with flags.

Regardless of the solution, we'd also need to update GOPRIVATE. I believe we can set it to * to keep it simple. Public repos are unaffected by this.

[mholt]

I could get behind a JSON config file for these more advanced configs...

In fact, the ability to describe a specific Caddy build configuration via JSON seems elegant.

[yroc92]

@francislavoie as for your proposed solution, that user@ syntax works for git commands, but not go get. So we can't indicate the user per repo with go get, which is what xcaddy is using to fetch plugins. But it does work when cloning, as I mentioned above.

You can have multiple accounts in a netrc file, can't you? I've never used it before but a quick Google suggests you should be able to do that.

https://stackoverflow.com/a/50771012

[francislavoie]

Okay cool 👍

[francislavoie]

So I guess the probable solution here involves having a JSON config as input as instructions to build.

It would take precedence over any --with flags if specified. Maybe --buildconfig buildconfig.json (and --buildconfig - support for STDIN I guess).

This would probably take some heavy refactoring; we'd need a new struct that the JSON unmarshals into, and we'd probably want to change --with to append into this build instructions struct, after a bit of string parsing.