Support private GitHub repos for gflows packages

jbrunton commented 3 years ago

For Github, this would require adding an auth header to requests to the raw.githubusercontent.com domain.

Suggested changes

See comment below for some further reasoning.

GFlows will check for a token. This token may be passed as a command line flag token, or an environment variable GFLOWS_TOKEN, the former taking precedence. (Or perhaps githubToken and GFLOWS_GITHUB_TOKEN to be clearer?)
If a token is given, all requests to packages hosted on githubusercontent.com will have an auth header added with the token.

Usage

For local dev, the user should ensure that the GFLOWS_TOKEN is set, either for their global shell, or using something like direnv.
For CI, the user should add the token to their GitHub secrets and use it in the gflows workflow, in the validation step:

- name: validate workflows
  env:
    GFLOWS_TOKEN: ${{ secrets.MY_PRIVATE_TOKEN }}
  run: gflows check

jbrunton commented 3 years ago

Mapping out the use cases:

1. Local dev

I probably want the token added to my shell automatically. I wouldn't want to have to manually run e.g. gflows --token <my-token> update every time.
In most use cases my GitHub user will have access to the private repo in question, so I can use a personal access token. Some users may already have a GITHUB_TOKEN or GH_TOKEN exported for other CLI tools.
I could also use something like direnv if I want more localized access.

2. CI

The simplest approach is to have the user amend the gflows workflow to specify the GFLOWS_TOKEN for the validation step.

An alternative would be to have the setup-gflows action set the GFLOWS_TOKEN env var based on its given token, but:

Is that potentially surprising for the end user? It is probably clearer to set GFLOWS_TOKEN explicitly in the step it is required in.
That would require a change to the workflow regardless, so why the indirection?

ghost commented 3 years ago

Hello @jbrunton!

Thanks for this idea, it is really good from my point of view.

Imho, gflows is a tool and should support CLI parameters rather than ENV variables.

What do you think about introducing "login" command like github CLI and docker CLI

for the minimal implementation, we could have login via github token only, like:

gflows login --token <my-token>
gflows update

It will allow users to access their private repositories with comfort from day 1.

1. Local development

Developers will login once between dev machine reboot, which should not be a big deal. Interested parties can export token to env variables and login to gflows as

gflows login --token $MY_PERSONAL_GITHUB_TOKEN
gflows update

or even add auto-login to bash profiles

for an advanced use case, credentials storage could be added for a better local development experience in the future. Like support for any git repositories beyond GitHub, storing credentials in a local config file, and so on.

2. CI

the proposed solution will not require additional improvements as it could be used as

- name: validate workflows
  env:
    GFLOWS_TOKEN: ${{ secrets.MY_PRIVATE_TOKEN }}
  run: |
   gflows login --token $GFLOWS_TOKEN
   gflows check

Alternative git-based approach

The other approach I can think of is to expose git dependency directly, and check out the whole package folder instead of accessing by direct file URL. In such a case authorization could be handled by git and gflows should just list an exact git command it is going to run in the documentation.

Sincerely, Andrey

jbrunton commented 3 years ago

Hi @andrey-covergo, thanks for the feedback!

On reflection, I think I favor your second suggestion, the git-based approach. This would fit nicely with the use case (which after all, is entirely about solving a GitHub problem), and avoids the need for implementing secrets management (which I'm keen to avoid, and would be redundant if we can simply rely on GitHub auth instead).

Just mapping out how this might work in more detail:

Packages could be given in terms of a git URL, e.g. git@github.com:my-org/my-repo.git (or git@github.com:my-org/my-repo.git/my-lib if the package is not at the root), possibly with a shorthand for github like github.com:my-org/my-repo/my-lib. In such cases, gflows would clone the repo instead of fetching files individually.
When developing locally, this will just work, assuming you have access to the repo.
On CI, I think you'd need to amend the default gflows workflow like below:

  steps:
    - uses: actions/checkout@v2
+     with:
+       token: ${{ secrets.SOME_PRIVILEGED_TOKEN }}
    - #@ setup_go()
    # etc.

I think this should be a pretty easy change to make. Let me know what you think.

ghost commented 3 years ago

Hello @jbrunton,

I think exposing a git dependency for the GitHub-specific tool is an acceptable decision from the design point of view, and also good in term of efforts.

We can take a look at other plugins to see how do they handle the same task -> access to a private git repository. For example, an action to download and attach your company private github actions to your workflow. It supports several methods of GitHub authentication. In our project we use GitHub app-based authentication, here is live example:

 - name: Checkout GitHub Action Repos
      uses: daspn/private-actions-checkout@v2
      with:
        actions_list: '["covergo/get-version@v1","covergo/get-issue-key@v1.1","covergo/docker-extract@v1","covergo/docker-diagnose@v1"]'
        checkout_base_path: ./.github/actions
        app_id: ${{ secrets.PRIVATE_ACTION_APP_ID }}
        app_private_key: ${{ secrets.PRIVATE_ACTION_APP_PRIVATE_KEY }}

As I know GitHub recommends using an app-based approach. However, for MVP of this feature, I suggest to pick the easiest way of authentication that is secure enough.

Sincerely, Andrey

jbrunton commented 3 years ago

@andrey-covergo: Thanks for the link. I haven't looked into how auth works with GitHub Apps, but certainly happy to investigate (especially if it's recommended).

I'll start with token auth, as I have a working prototype and should be able to get a PR merged fairly soon, and will then open a subsequent issue to discuss details of app based auth.

jbrunton commented 3 years ago

@andrey-covergo: First iteration of this is now available. In the end, I went with SSH for both local and CI purposes to get started. I updated the docs here: https://github.com/jbrunton/gflows/wiki/GFlows-Packages#git-and-private-packages

Lmk if that's good enough to get started.

ghost commented 3 years ago

Hello @jbrunton, thank you so much for working on this ticket!

Unfortunately, I'm getting an error during the local development cycle, when run "gflows update":

andrey@Andrejs-MacBook-Pro Auth % ./gflows update
Cloning git@github.com:CoverGo/gflows.git...
Enumerating objects: 167, done.
Counting objects: 100% (167/167), done.
Compressing objects: 100% (143/143), done.
Total 167 (delta 70), reused 106 (delta 9), pack-reused 0
open /var/folders/bz/dxvjbb251ms5scjplkpcfzrw0000gn/T/127022773/v1.6/.gflows/gflowspkg.json: no such file or directory

My config file:

# Config file for GFlows.
# See https://github.com/jbrunton/gflows/wiki/Configuration for options.
githubDir: .github
templates:
  engine: ytt
  defaults:
    libs: 
    - local-libs 
    dependencies:
   #  - https://raw.githubusercontent.com/CoverGo/gflows/v1.6/.gflows
     - git@github.com:CoverGo/gflows.git/master/.gflows

target repository

jbrunton commented 3 years ago

@andrey-covergo: The gflows implementation assumes that the files will be on the main branch of the repo, so you don't need to specify that. The format is basically <git-repo>/<local dir>, and in your case I think you therefore want git@github.com:CoverGo/gflows.git/.gflows.

(Also, to double check, since I notice you committed the gflows binary, this will only work with gflows 0.4.)

This does the raise the question: what if you want to test against a dev branch? It would definitely be useful to support a branch option for git repos, but that is not yet implemented. I created https://github.com/jbrunton/gflows/issues/195 to track.

ghost commented 3 years ago

@jbrunton thanks for the comments! git@github.com:CoverGo/gflows.git/.gflows works fine, I've just checked it. I assumed branches and tags should be supported and just copied a part of URL from HTTPS dependency. Binary in the repo is old and I should update it, and it is used only for template testing.

I can say this ticket could be closed as now we have #195 for branch support.

Really appreciate your efforts!

jbrunton commented 3 years ago

@andrey-covergo: Glad it's working, thanks for the feedback! I will go ahead and close this.

jbrunton / gflows