search

woodpecker-ci / woodpecker

Woodpecker is a simple, yet powerful CI/CD engine with great extensibility.
https://woodpecker-ci.org
Apache License 2.0
4.29k stars 369 forks source link

Gerrit Forge #4232

Closed pelme closed 2 weeks ago

pelme commented 1 month ago

Clear and concise description of the problem

As a user of Gerrit Code Review, I would like to use Woodpecker for CI. Opening this issue to see if there would be any interest in having this in core Woodpecker as a supported forge or if it would be a better approach to add it as an addon forge.

Guidelines for adding new forges

As documented (https://woodpecker-ci.org/docs/development/core-ideas#guidelines), a to add a new forge to Woodpecker, it must support webhooks and oauth2:

Webhooks

Gerrit supports webooks via the builtin core plugin webhooks: https://gerrit-review.googlesource.com/Documentation/config-plugins.html#webhooks.

OAuth2

Gerrit is not an OAuth2 provider itself, does that disqualify it from being added as a core?There is a popular Gerrit plugin to make Gerrit use OAuth: https://gerrit.googlesource.com/plugins/oauth/ In our setup, we use Google SSO OAuth to authenticate with Gerrit. So we would use Google SSO with Woodpecker too.

Would it be possible to reconsider the OAuth2 requirement? Or make it possible to have another OAuth2 provider while still having Gerrit support in core Woodpecker?

Suggested solution

Add Gerrit as a builtin forge.

Alternative

Build a Gerrit integration as an addon forge.

Additional context

My company would potentially be happy to open source/contribute code and/or money towards getting Woodpecker/Gerrit working somehow. Of course we could start building it as an addon forge.

Gerrit is a great tool for git+code review and a lot of people like that flow as an alternative to the typical PR workflow Github/Gitlab/Gitea etc. Gerrit is open source and has been around for a long time and is actively developed. Using Gerrit and Woodpecker together seems like it would be an amazing combination for a lot of people! 🔥

Both Gerrit and Woodpecker uses birds as their logos/mascots! 🐦‍⬛

Validations

pelme commented 1 month ago

Just found this PR. A long time ago Gerrit was a placeholder in the supported features docs: https://github.com/woodpecker-ci/woodpecker/pull/372 😆

zc-devs commented 1 month ago

Even though I do not need Gerrit support, you are rising interesting questions, one of them I also thought about.

it must support webhooks and OAuth2

OAuth matters (#2161, #3591), but not webhooks.

Or make it possible to have another OAuth2 provider while still having Gerrit support in core Woodpecker?

That is the most interesting part to me. Let me rephrase: can we use arbitrary OAuth provider for authentication and different supported forge for repositories stuff? For example, I use Gitea as SCM. I set it up to use external OAuth provider, for example Keycloak. And currently it looks a bit weird: WP login -> redirect to Gitea -> redirect to Keycloak -> and then in reverse direction (I guess). It would be nice if Woodpecker could use Keycloak directly for authentication and Gitea for repositories stuff.

Edit 1 Perhaps, WOODPECKER_EXPERT_FORGE_OAUTH_HOST would make it possible. Despite, Forge interface should be split, I think.

being added as a core?

Interesting discussions are #2651 and #2355.

Of course we could start building it as an addon forge

You might be interested in #3266 then.

pelme commented 1 month ago

Thanks for you reply!

Splitting the Forge interface to be able to use any OAuth2 plugin/provider with any Forge seems like it would be reasonable. Would mapping a user from any OAuth2 backend to a user in a forge be a problem? Or is it always mapped via the email address?

The use case you describe with Gitea/Keycloak would be improved if Woodpecker could go directly to Keycloak.

For Gerrit users, it would also be very helpful. Gerrit is a central git server and hosts the equivalent of PRs. I think many Gerrit setups is used with another forge/gitweb such as github/gitlab/forgejo to get code browsing and issue tracking. In those cases, being able to reuse the existing OAuth2 backends already present in Woodpecker seems like a big win. Gerrit/Forgejo/Woodpecker with Forgejo being the OAuth2 provider could be a lovely setup. Splitting the interface into OAuth2/Forge parts would make that possible.

(I just found out Woodpecker a couple of days ago, am not familiar with the code and do not know Go so I may be missing a lot of context here! 😬 )

zc-devs commented 1 month ago

am not familiar with the code

Me almost too :) except Agent's Kubernetes backend.

Gerrit/Forgejo/Woodpecker Splitting the interface into OAuth2/Forge parts would make that possible.

I used to use it at one of previous job. ~Though, I forget it's just review system. Thank you for reminding me.~ So, Forge interface should be split into ~3 pieces then: authentication, repository and pull requests (for example, Keycloak, Forgejo, Gerrit respectively).~ 2 pieces: authentication (Forgejo in your example) and repo stuff (Gerrit in your example).

~That being said, I'm sceptical about chances to get it in the "core" ;)~ It's doable then.

qwerty287 commented 4 weeks ago

I think we should not split this. As @pelme said, that would increase make it much harder to connect the users to the forge users I think. Remember that we use oauth to get a token from the forge and we need that one. I'm not an oauth expert, but I don't think this is easily doable with a direct authentication via an external oauth backend.

OAuth matters (#2161, #3591), but not webhooks.

Generally speaking, yes, but I think it's a bad idea to add something without webhooks to the core as it could be confusing to users. That's why we introduced add-on forges. They can implemented almost exactly as in the core, you can publish the source code and anyone else is able to use it too.

So, about gerrit: Missing oauth is a problem as I said, so I'd rather go with an add-on forge for this.

In case you use gerrit with forgejo/github etc, why don't you use woodpecker and connect it to this forge?

zc-devs commented 4 weeks ago

In case you use gerrit with forgejo/github etc, why don't you use woodpecker and connect it to this forge?

Yeah, it was messed me too :) but initial request was Google SSO + Gerrit. So OAuth (Google, Amazon, Azure, Keycloak, etc) + Forge (Gerrit, Gitea, etc).

I don't think this is easily doable with a direct authentication via an external oauth backend

I'll check that using WOODPECKER_EXPERT_FORGE_OAUTH_HOST. Anyway, this is a feature request from me then :)

I should have noted, that Woodpecker and Forge have to use the same provider, but it is not a problem in enterprises, I think, and definitely not pelmes case, nor my example.

I think we should not split this ... I'd rather go with an add-on forge

Addon doesn't solve the problem. If pelme implemented current Forge interface as Addon, then he would end up with binaries implementing combinations of authentication and repo like Google+Gerrit, Keycloak+Gerrit and so on.

PS There is a char in SOLID responsible exactly for that problem. Addons is a joke to me :)
qwerty287 commented 4 weeks ago

Sorry @zc-devs i just deleted your comment accidentally.

If I remember correctly it was something like:

I'll check that using WOODPECKER_EXPERT_FORGE_OAUTH_HOST .

Each forge has AuthURL and TokenURL. Using the env var you can change the auth URL, but token URL is unchangeable so testing doesn't make sense.

So do you think it is possible to just change the token URL to solve the problem? How would that work then? WP gets a login token from the oauth provider and using this token it can query the forge to get a forge token?

pelme commented 4 weeks ago

To clarify responsibilities when I said you could use Gerrit/Forgejo/Woodpecker together:

If we talk about splitting the Forge interface in three (or more) it would be GitRepo, ChangeRequests, GitWeb, OAuth2Provider. I do not know if it makes sense. But it would not be a problem for the existing forges since they would all implement the parts anyways?

I do think that a addon forge would be a good start before considering it for core. A addon forge implements exactly the same interface and can do provide exactly the same functionality as the builtin ones, right?

zc-devs commented 4 weeks ago

It was

I'll check that using WOODPECKER_EXPERT_FORGE_OAUTH_HOST

Each forge has its own AuthURL and TokenURL. It likely be different for Keycloak/Google/etc. While we can override AuthURL by WOODPECKER_EXPERT_FORGE_OAUTH_HOST, we cannot do the same with TokenURL. So, it doesn't make sense to test it in practice, until we can override TokenURL.

So do you think it is possible to just change the token URL to solve the problem?

I just gave up on testing in practice. Anyway it seems the first step to support an auth provider, separated from a forge.

WP gets a login token from the oauth provider and using this token it can query the forge to get a forge token?

What is the difference between this tokens? Is "provider" token like ID-token, which contains only profile (login, email), but "forge" like authorization, which contains permissions (like admin on this repo and contributor on another)?

How would that work then?

I login in Gitea using Keycloak, got a token, can use Gitea.

I login in Woodpecker using the same Keycloak, got a token. Can use Woodpecker's own functionality. But Woodpecker calls a forge. How would that work - is that your question? I assume, for that purpose forge token is used. Could we issue a token compatible with Gitea at WP login? So, when WP calls Gitea it passes through the user's token. It should be possible to issue one, which will work for Woodpecker/Gitea at the same time.

Above, I considered user interactions. But what's about the webhooks, crons? So, WP should execute some pipeline at some time. What token will WP use to call a forge? The same is for webhook. Is forge token used for this purposes? I wonder, how it is implemented now.

Yeah, overall it seems like a difficult question. I appreciate if someone shed light on that. Edit: there. I suppose, it should be possible to issue a forge-compatible token for an "initiator"-user (or bot) directly from an auth provider and then refresh it directly also.

Forgejo: Repository browsing

You can browse the code, see the graph, blame in the IDE, but we do not discuss it here. So, I propose to cross it off the discussion. It was confused me (with 3 parts splitting instead of 2) and qwerty. Thanks for refining, though.

qwerty287 commented 4 weeks ago

What is the difference between this tokens? Is "provider" token like ID-token, which contains only profile (login, email), but "forge" like authorization, which contains permissions (like admin on this repo and contributor on another)?

You have two different softwares running, let's say forgejo and keycloak. If you log in via keycloak, you probably won't get a token usable with the forgejo api. We need a token to query the forge apj though. The first token from the login process (the keycloak token in this case) is not important for woodpecker. But we need the forgejo token and tbh I don't see a way to get this token.

But Woodpecker calls a forge. How would that work - is that your question? I assume, for that purpose forge token is used.

Yes, that's what I mean.

Could we issue a token compatible with Gitea at WP login? So, when WP calls Gitea it passes through the user's token. It should be possible to issue one, which will work for Woodpecker/Gitea at the same time.

So you mean that the user manually has to set a token? That would be really bad ux-wise.

Above, I considered user interactions. But what's about the webhooks, crons? So, WP should execute some pipeline at some time. What token will WP use to call a forge?

The point is: currently there's just one token because the authentication system is the forge . WP needs to get a token to call the forge from the authorisation provider. Without having looked into this detailedly, I don't think that's possible.

The same is for webhook. Is forge token used for this purposes? I wonder, how it is implemented now.

The forge token is needed to create the webhook (on repo add/repair), otherwise not directly. Woodpecker stores the user's forge token in it's DB and then uses this token for further calls after a webhook.

I suppose, it should be possible to issue a forge-compatible token for an "initiator"-user (or bot) directly from an auth provider and then refresh it directly also.

I don't understand this. Do you mean you manually set a forge token for a bot user in you wp config?

zc-devs commented 4 weeks ago

https://developer.okta.com/blog/2019/10/21/illustrated-guide-to-oauth-and-oidc https://www.keycloak.org/securing-apps/oidc-layers https://openid.net/specs/openid-connect-core-1_0.html

The user ID token will be the same and valid for all systems. For example:

{
  login: john,
  email: john@consto.com,
}

The user access token... well, it depends... each system requires own claims, but in theory we can issue universal access token like:

{
  gitea_roles: admin,
  woodpecker_roles: user,
  consto_roles: developer
}

and use this token in any system. https://stackoverflow.com/questions/53687156/managing-application-access-with-openid-connect-and-oauth

So you mean that the user manually has to set a token? Do you mean you manually set a forge token for a bot user in you wp config?

No. I imagined schema like Gitea -> Keycloak <- Woodpecker. Woodpecker do not call Gitea for tokens at all. Keycloak gives universal token for both WP and Gitea (see example above).

However, I've just checked interactions with Gitea and didn't see Authorization header in requests, but cookie i_like_gitea. Seems, Gitea uses OIDC provider only for authentication. Even though it can distinguish admins and restricted users Screenshot 2024-10-18 1, but I have to supply this claim in ID token.

Besides, I've found the scopes used in Gitea. Seems, Woodpecker requests Gitea-specific token with all scopes.

Even if it is possible to issue an access token in Keycloak with Gitea's claims/scopes, it doesn't make sense. @qwerty287, thank you for answers and leading questions. I revoke my this is a feature request from me then :)

Then the question is how Gerrit works? Does it issue custom token to access API or entirely rely on scopes from OIDC? That are the questions for you, @pelme. Wish you luck and looking forward to implementation.

qwerty287 commented 2 weeks ago

@pelme Did you check out addon forges? As explained, we probably won't support gerrit natively.

pelme commented 2 weeks ago

Not yet, still looking at alternatives. I agree it seems not feasible to support gerrit natively at this point. Addon forges is nice, when/if myself/someone else figures out more details and actually build a gerrit forge addon I guess it can be discussed again. 🙂

qwerty287 commented 2 weeks ago

Ok, so we can close this? If you need any help for your addon, just ask

6543 commented 2 weeks ago

I personally think, if wp would be able to intergrate to gerrit would be a huge win for the project ...

... but I dont want to add it to the core of the base requrements of a forge (to us) are not met.

So I would say I personally would help with looking at an addon and somethimes do stuff there ... but starting such addon is currently not in my time budget

so - I'm also for closing this issue but we still are happy to stay in contact to anyone who wana move that front forward ;)

cc @pelme

PS: note to me: a generic addon that pulls git(/...) and emulates an forge might be a good generic way like a bridge to onbord all strange types of forges ...

6543 commented 2 weeks ago

just reread the issue and found out about https://gerrit.googlesource.com/plugins/oauth/

well I would argue it's valid to add it to the core, and requre an addon on the gerrit side ...

... still we currently have no maintainer who use gerrit so I dont know how would implement&maintain it

qwerty287 commented 2 weeks ago

@6543 please read the comments in this issue. We figured out already that this won't work.