GitHub App as authentication method

[akselleirv]

I need to have more fine-grained access control, as Kargo requires RW to a Git repo, than the use of PAT.

From my understanding reading the docs that this is currently not been implemented. Is this something that will be accepted as a contribution?

I can take inspiration from how ArgoCD has done it: https://argo-cd.readthedocs.io/en/stable/user-guide/private-repositories/#github-app-credential

[krancour]

@akselleirv thanks for the interest in Kargo!

Without dismissing your idea, I am curious to first know more about this:

I need to have more fine-grained access control, as Kargo requires RW to a Git repo, than the use of PAT.

Are you able to be more specific about what you cannot currently achieve using a PAT? It seems to me that the newer "fine-grained tokens" that are in beta (as opposed to the "classic" tokens) offer a pretty significant degree of control over what a PAT can and cannot do.

[akselleirv]

Thanks for the quick reply @krancour!

I tried to create a new fine-grained PAT, but I was not able to create one for a repository within my organisation. Only for my own personal projects.

Another reason was that I would like to create a GitHub App as that is not connected to a user.

The last reason is that it would possible to let users in my organisation to opt-in/install the app to their repository. So the access is limited to only those who request it.

[krancour]

I tried to create a new fine-grained PAT, but I was not able to create one for a repository within my organisation.

That's just a matter of permissions. You can not delegate authority that you yourself do not have.

Another reason was that I would like to create a GitHub App as that is not connected to a user.

That makes sense.

Mainly on account of this, I'll leave this open as something we can take under consideration when time permits.

The last reason is that it would possible to let users in my organisation to opt-in/install the app to their repository. So the access is limited to only those who request it.

Fine-grained PATs do address this. You can limit the scope of the PAT to specific repositories.

[joebowbeer]

@krancour GitHub App auth is a requirement in our org, for security reasons

I've gathered some justifications and pointers to prior work below

According to GitHub's documentation, GitHub recommends using a GitHub App:

If you want to use the API on behalf of an organization or another user, GitHub recommends that you use a GitHub App.

https://github.com/jenkinsci/github-branch-source-plugin/blob/master/docs/github-app.adoc

Why?

• the rate limit for a GitHub app scales with your organization size, whereas a user based token has a limit of 5000 regardless of how many repositories you have,
• for organization’s that have 2fa enforced - no need to manage 2fa tokens for a 'bot' user
• to improve and tighten security: the Jenkins GitHub app requires a minimum, controlled set of privileges compared to a service user and its personal access token which has a much wider set of privileges

Support was added to ArgoCD in early 2021:

https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#github-app-repositories

https://github.com/argoproj/argo-cd/issues/3086

GitHub Apps also provide a far greater rate limit, have fine grained access control and as a huge plus, you don't even need to manage users for it.

https://github.com/argoproj/argo-cd/issues/4348

[github-actions[bot]]

This issue has been automatically marked as stale because it had no activity for 90 days. It will be closed if no activity occurs in the next 30 days but can be reopened if it becomes relevant again.

[joebowbeer]

/not-stale GitHub App is still the recommendation

[krancour]

Making a note for myself to tackle this in tandem with the solution to #1885

[krancour]

Actually, this will be solved separately from #1885, but will be solved in a consistent manner.

[krancour]

@jessesuen I have tentatively moved this to v0.9.0. Please feel free to move it back if/when you determine that this is equal to or greater than #1680 in priority.

[krancour]

It is moved back to v0.8.0.

[krancour]

Adding some notes here to allow prospective users of this feature to comment.

The ideal I wish were achievable here would be registering a single GitHub App per Kargo instance. Users would install that one GitHub App wherever that Kargo instance needed repository access. When acting on behalf of a given project, Kargo would obtain app installation access tokens to be constrained to Project-specific privileges.

This would work very well if every Kargo Project (or every team, at least) had its own GitHub org, but excluding those using GitHub Enterprise, that's an unlikely scenario. For a company with just one (or a handful) of GitHub orgs, the notion of each team installing the one Kargo App independently of one another, while granting access only to their own repos, will be entirely out of reach.

This will leave most orgs with two choices:

1. One App, installed once, into one org == same permissions for every Kargo Project. This violates the principle of least privilege.

2. Register a separate GitHub App per Kargo Project (or per team, at least) and install those separate Apps, once each, into the one org, with each installation granting permissions on specific repositories. GitHub orgs are limited to 100 App registrations each, which means this option scales poorly.

Given these limitations:

1. Does anyone know of options we are overlooking?

2. Do the limitations alter the feature's value proposition for those who have been anticipating it?

[joebowbeer]

@krancour asks

Do the limitations alter the feature's value proposition for those who have been anticipating it?

Short answer: No

Why is this a more difficult sell than it is for ArgoCD, which implemented this 3 years ago, or Flux, which is implementing it now? Or Backstage, which is typically used by many teams?

We have a few teams using a few GitHub repos (only the few specifically devoted to GitOps) that are synced by ArgoCD using a single installation of our GitHub App. We'd like to add Kargo to the mix, preferably or at least initially using the same GitHub App that we use for ArgoCD GitOps.

We only grant read access to our GitOps repos, so it's no problem to add a list of repos to our GitOps GitHub App.

Even if we allowed write access to some GitHub repos, don't these Kargo projects enforce their own RBAC? Why should the GitHub App be responsible for enforcing project-level policies?

ArgoCD has projects, too. Do they raise the same concerns that Kargo projects do?

[krancour]

Why is this a more difficult sell than it is for ArgoCD

No one is resisting doing this. We're just asking if those most eagerly anticipating this feature, which seems to include you @joebowbeer, understand and accept its limitations. Or maybe some may have ideas for working around those limitations.

We only grant read access to our GitOps repos, so it's no problem to add a list of repos to our GitOps GitHub App.

I suppose this is a difference worth calling out. You may feel comfortable giving most/all Argo CD projects/Apps read-only access to a very broad swath of repos, but since Kargo also writes to repos, you might not feel so willing to not enforce the principle of least privilege more rigorously.

don't these Kargo projects enforce their own RBAC? Why should the GitHub App be responsible for enforcing project-level policies?

Kargo uses pure k8s RBAC. Repos are not k8s resources. If you have one App and you install it into one org and you give it broad access to lots of repos and then all your projects use that one App installation to authenticate, all the projects get the same privileges on those repos.

Argo CD's RBAC works differently.

[taer]

For us, I used a single GH app that has RW access to a handful of "deploy repos". We have 6 top level teams, and each one has it's own deploy repo. Kinda a semi-mono-repo. They might have N apps each, but they all get deployed from their teams repo.

I used a RW app in anticipation of either argo image updater or Kargo.

So an option for a single app for all would be perfect for us. The installation of the app "approves" the repos it can write to, so for our use case, adding 6 RW repos to a single app that kargo uses is pretty safe I think

[joebowbeer]

A couple of observations, questions:

I was under the impression that write access was an option in Kargo. This is an area I want to suss out more, but it doesn't really affect our interest in this auth method.

Your comments refer to "lots of repos" and we do have lots of repos, but we only use a very few for GitOps. We started with one GitOps monorepo, dedicated to ArgoCD Applications and their manifests, and have added a couple more after ArgoCD supported multiple sources. But the hundred or so source code repos which build and publish the images are not accessed by ArgoCD.

[krancour]

Write access is needed for Stages that use Git-based promotion mechanisms. If your promotions do not involve writing anything to a Git repo, then read is sufficient.

Re: lots of repos -- sorry. To clarify, I'm only ever referring to gitops repos.

So to rephrase more accurately: If you have one App registered and it asks for certain permissions wherever it is installed, and you install it to one org and, and as part of the installation, select a subset of repos, anyone/anything using that App installation to authenticate will all have identical access to the selected subset of repos.

And this makes me realize another limitation...

Desired permissions are defined in the App registration. An installation gets those permissions on all repos selected when it is installed. If you happened to need read/write on some of those, while read-only is sufficient on others, there's actually no way to accommodate that. You end up stuck having to do r/w on all of them and that's another principle of least privilege violation. Not sure there's much that can be done about that.

The only thing I am really getting at is there are some less than ideal aspects to using GitHub Apps for authentication and suggestions about how to mitigate them are welcome.

[jessesuen]

I wanted to summarize the discussion a bit, since it may be hard to follow through the multiple comments.

GitHub App authentication will be a feature in Kargo, slated for v0.8. However, our research has revealed limitations with GitHub App Authentication that users should know about since it is not a panacea and will not work for everyone.

Some background:

First, Kargo is typically given git write permissions to facilitate a GitOps promotion. Argo CD never needed this privilege; it only needed read. Because git write affects what gets deployed, it's also a much more dangerous privilege to manage.

Second, Kargo Projects are intended as a hard tenancy boundary (much harder than Argo CD). Under the covers, Kargo Projects are just Kubernetes Namespaces. A user who can supply working credentials into their own Project Namespace is free to use it, and other Projects cannot use/access it. This works great because it enables self-service (e.g. Team A can input their git write credentials into Project A. Team B can do the same for Project B). This self-service is an improvement over Argo CD, which required an Argo CD Project admin to allow-list the git repos used in the projects that they vended to others.

The challenge with GitHub App Auth is you have to decide if you want:

1. A single application registration, with write access to multiple git repos, where these git repos might span multiple Kargo Projects (tenants).
2. A different application registration per Kargo project. Where each registration has write access to a single git repo.

The problem with option 1 is the loss of the principle of least privilege access. Basically, it results in a single credential that can write to all repos (referenced in the GitHub app). Ultimately, this means that Team A can simply reference a git URL belonging to Team B and perform gitops changes to Team B's repo. This may be manageable in small organizations with a high degree of trust between sister teams, but it would not be acceptable in larger ones.

The problem with option 2, is that a GitHub org is limited to 100 app registrations. So if one GitHub App is registered for each Kargo Project, it means at most 100 Kargo Projects could use this technique. Finally, unlike PATs, app registrations require a GitHub org admin to approve, so the end-to-end process of supplying GitHub App auth to a project, is not exactly fully self-service.

With all that said, none of the above may be huge concerns if the number of GitOps repos that Kargo needs to write to is limited (e.g. you are using GitOps mono repos, or simply do not have many services/Kargo Projects). But it would be a factor when there are many GitOps repos and granular permissions to write to individual repos are needed.