Proposal: Support git providers other than GitHub in prow

[wbrefvem]

What would you like to be added:

Some sort of generic git service in prow that normalizes incoming webhooks into a generic format that hook and other plugins can understand, as well as provides a generic interface for performing git-related actions such as commenting on PRs, etc.

My initial proposal is that this git service essentially be another plugin that implements a number of generic git-related APIs, and that the name/location of this service be configurable so that we can support out-of-tree providers. In addition, it's probably worthwhile to think about decomposing the notions of code review and issue tracking into separate provider APIs, since not all providers implement these functions, although presumably prow will only ever be useful in combination with a git provider that at least provides basic code review features. But it would be nice to support, e.g., code review tool X in combination with issue tracker Y. We currently have something like this with the loose coupling of the Gerrit adapter and reporter. It would be great if we could make this sort of thing generic and extensible.

We discussed this briefly on the sig-testing call of November 6, and my hope is that this issue will be an asynchronous continuation of that discussion. I am also working on a more formal proposal that I plan to present on the weekly sig-testing call some time in the near future.

Why is this needed:

Currently, prow is tightly coupled to GitHub. Making it more generic would give the various k8s projects more flexibility in how they craft their workflows, as well as making prow more accessible to projects in the broader open source community and beyond.

/kind feature

[BenTheElder]

cc @cjwagner @krzyzacy /area prow

[yastij]

we might want to think about an interface that providers might want to implement.

[wbrefvem]

@yastij Yeah, there seems to be some consensus around this.

[wbrefvem]

Update

Here are some more concrete ideas on how to tackle this:

• Implement a git service as a freestanding gRPC server
• At least one webhook normalization method that would transform an incoming webhook into a generic event that hook can dispatch
• Individual methods for various actions such as AssignIssue, CommentOnPR, etc.
• Ship a default service that supports popular providers such as, but not necessarily limited to, GitHub, GitHub Enterprise, GitLab, Bitbucket.
• Make protobuf package and .proto files available for users who want to implement their own service.

[midnightconman]

This PR https://github.com/kubernetes/test-infra/pull/11075 was just merged, it gets us closer to this goal.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[yastij]

/remove-lifecycle stale /lifecycle frozen

[townsendb]

Hi @BenTheElder . Is this proposal on the radar? I'm using GitLab and trying to use the next gen serverless features of Jenkins-x but blocked by this issue in prow. Many thanks.

[McShauno]

Ok, fine ... I'll write it. Give me a couple of weeks.

¯_(ツ)_/¯

[BenTheElder]

@townsendb Hi, Sorry I missed this comment in all my github notifications somehow... I don't really work on prow currently (kind mainly) though I'd love to see this happen. I'm not sure where this is at.

@cjwagner & @stevekuznetsov are good current points of contact for prow.

[BenTheElder]

I think there was a more specific proposal related to this for the proposed "scalleywag" component but I'm not sure where that's at.

[polothy]

Just for those not aware, scallywag is here https://github.com/kubernetes/test-infra/pull/12241 and part of it was merged here https://github.com/kubernetes/test-infra/pull/12700

[berbiera]

Hi there, Is there a chance to get this proposal accomplished ?

[bgrant0607]

Previous request for gitlab: #8435

(connecting dots to make this issue easier to find)

[drhender]

I am also attempting to integrate prow with our internal deployment of Gerrit, but the limitations described in this issue prevent me from doing so.

To be clear, we use internal deployments of K8s and Gerrit-- not something deployed on a public cloud. Without support for such an environment, Prow (and Argo and Knative and Tilt and a whole host of other "K8s native" CI/CD tooling) is of limited use.

[BenTheElder]

@drhender it's worth noting that Prow unlike argo, knative, tilt etc. was originally written out of needs for the kubernetes upstream project to do it's own testing, rather than as a general tool. It was never expected to work in private environments and I think that's reasonable as opposed to knative, which hopefully does!

Now, Prow actually does have limited support for private Gerrit specifically at the moment, so it might be worth looking into that a bit more. I know this is actually in use.

[stevekuznetsov]

@drhender so the issue here is about supporting GitHub-like git providers like GitLab, IMO. Right now gerrit workflows do have support but since gerrit provides a very different UI and workflow than the git/issue/tracking/whatever solutions like GitLab or GItHub, not every part of Prow is supported on gerrit. What issues are you running into?

[whiskerch]

Is there any update on this? We are using prow on our projects, but need to set up some new ones on an isolated network - we're going to host our own gitlab instance as the cost of a self hosted github instance is prohibitive

[deedubs]

I'm interested in taking a look at adding gitea support where are we with regard to implementing @wbrefvem's suggestion of gRPC vs adding the additional services in tree

[RafalMaleska]

What is the status of this issue?

I was trying to use Lighthouse for Prow integration into Gitlab - it seams that it provides only basic prow functionality and requires JenkinsX installation.

Is there a RoadMap to develop Prow further?

[alvaroaleman]

What is the status of this issue?

If someone is interested in working on there will be progress.

Is there a RoadMap to develop Prow further?

No, prow is mostly developed by the ppl that use it according to their needs

[abdennour]

we love prow. But, we are using bitbucket cloud. Any change ?

[cjwagner]

@abdennour No change, Alvaro's comment is still accurate. PRs are welcome if you'd like to contribute to this!

[spiffxp]

/sig testing

[vsoch]

I'm definitely interested in this use case (and new to prow but looking to try it out!) At the labs we use GitLab heavily, and it would be great to have this integration. The degree to which I could help will probably come down to if it's easy to set up a development environment I think.

[vsoch]

Also the README says it was going to be deleted in February but... it still lives!! cue Frankenstein monster noises

[BenTheElder]

There really need to be more concrete proposals, this thread is pretty vague.

prow has many aspects and most of them are inherently GitHub specific (eg everything related to webhook handling and label based chatops) which theoretically could be abstracted away, but at this point would take a lot of work and require a more detailed proposal

[vsoch]

I'm not familiar with how proposals are done here - akin to a KEP?

I was reading here: https://docs.prow.k8s.io/docs/life-of-a-prow-job/

And at least from a high level, the abstractions like hooks, comments, are things that can be mapped to other code review places too. I definitely don't disagree about it being a lot of work, but (at face value) doesn't seem impossible?

@BenTheElder are these the best docs https://docs.prow.k8s.io/docs/build-test-update/ for a developer to get started trying things out?

[BenTheElder]

I'm not familiar with how proposals are done here - akin to a KEP?

Depends on the scope of the change, typically a lightweight google doc is enough to start discussion, which should be shared to the SIG in the mailinglist and slack.

And at least from a high level, the abstractions like hooks, comments, are things that can be mapped to other code review places too. I definitely don't disagree about it being a lot of work, but (at face value) doesn't seem impossible?

It's not impossible, but it's a lot of work and as-is we're having difficulty getting much more scoped changes the Kubernetes project itself would benefit from reviewed / approved promptly, there's limited bandwidth from the current prow OWNERS.

@BenTheElder are these the best docs https://docs.prow.k8s.io/docs/build-test-update/ for a developer to get started trying things out?

I don't work on prow itself often anymore and when I do I make small fixes that don't require spinning up an instance.

[vsoch]

Interesting - I would guess there is a long history behind this, but why does prow have to live under Kubernetes here proper? If it’s holding the project back and adding extra churn for developers that doesn’t make a lot of sense.

[BenTheElder]

Interesting - I would guess there is a long history behind this, but why does prow have to live under Kubernetes here proper?

Prow exists because Kubernetes needed some CI tooling, which started as a webhook system for commands against jenkins and then a merge robot. It eventually turned to running CI workloads on Kubernetes itself.

Usage for other purposes beyond Kubernetes came much later. It still primarily exists for Kubernetes and supports workflows used by the project.

If it’s holding the project back and adding extra churn for developers that doesn’t make a lot of sense.

That's not what I meant. I meant that the existing owners are currently already struggling to keep up with incoming feature work so I'm warning that it may not be a great time to propose a very large change.

[vsoch]

Gotcha, and thank you for the details! It could be then that a better avenue to pursue would be creating something like prow, but separate from it and with a focus on more general abstractions for components.

If you (or anyone in this thread) know if other tools in this space that are for creating your own CI/CD on Kubernetes please share! When you do a general Google search it typically shows articles about CI/CD with respect to continuous deployment to Kubernetes, and not making your own system (that could work with it).

[kfox1111]

There was a fork of Prow I think a while back that eventually became Lighthouse? https://github.com/jenkins-x/lighthouse/

I wanna say it was intentionally done to try and add more then github support while not burdening the k8s developers with features they didn't need.

Might be what your looking for?

[BenTheElder]

I don't think we've had the bandwidth for this, but in any case prow is now split out to github.com/kubernetes-sigs/prow and future feature discussions should be held there.