search

tektoncd / triggers

Event triggering with Tekton!
Apache License 2.0
557 stars 419 forks source link

Polling a repository to detect changes and trigger a pipeline #1168

Open sm43 opened 3 years ago

sm43 commented 3 years ago

To Poll a repository at a certain interval to look for any changes and trigger a pipelinerun/taskrun.

Use cases:

This can be done using a cronjob but could to a part of triggers. Similar Issue: https://github.com/tektoncd/triggers/issues/480

Proposal: To have this feature as a part of triggers. I did a poc around this where I have a new SyncRepo CRD

apiVersion: triggers.tekton.dev/v1alpha1
kind: SyncRepo
metadata:
  name: test
spec:
  repo: https://github.com/tektoncd/hub
  branch: main
  frequency: 3m
  binding: pipeline-binding
  template: pipeline-template

Approach:

Things can be improved:

Available solution: There is an operator by Kevin https://github.com/bigkevmcd/tekton-polling-operator which polls a repo and create a pipelinerun/taskrun if something is changes. This is independent of TektonBinding and TektonTemplate, it take PipelineRef as input.

cc @vdemeester

sm43 commented 3 years ago

@dibyom @wlynch @savitaashture @khrm let me know what do you think? :upside_down_face:

vdemeester commented 3 years ago

We should also probably support "any" git repositories (and not only github/gitlab/…).

dibyom commented 3 years ago

We discussed this at the last WG and decided that the next steps would be to open up a TEP with the problem statement/use cases so that we can gather more requirements/feedback for this feature.

bigkevmcd commented 3 years ago

I was thinking about this, part of the complexity of this is integrating with Triggers and TriggerTemplates, but maybe it shouldn't, and instead, when it sees a new commit, it should send the commit as a "hook" to an EventListener, it would be "internal" ?

dibyom commented 3 years ago

but maybe it shouldn't, and instead, when it sees a new commit, it should send the commit as a "hook" to an EventListener, it would be "internal" ?

that is definitely an approach we should consider!

tekton-robot commented 2 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale with a justification. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

tisaiful31 commented 2 years ago

To Poll a repository at a certain interval to look for any changes and trigger a pipelinerun/taskrun.

Use cases:

This can be done using a cronjob but could to a part of triggers. Similar Issue: #480

Proposal: To have this feature as a part of triggers. I did a poc around this where I have a new SyncRepo CRD

apiVersion: triggers.tekton.dev/v1alpha1
kind: SyncRepo
metadata:
  name: test
spec:
  repo: https://github.com/tektoncd/hub
  branch: main
  frequency: 3m
  binding: pipeline-binding
  template: pipeline-template

Approach:

  • This looks for changes on repository using GitHub Rest API and triggers a pipelines.
  • The API returns the response in json format for example or https://api.github.com/repos/sm43/hub/commits/tekton-ci
  • It uses trigger binding to get values from the json response and use them in the trigger template and triggers the pipelinerun.
  • Later it saves the last commit on which it was triggered and use it in next reconcilation to look for changes.

Things can be improved:

  • Support for private repositories
  • Gitlab also have similar API which we can use
  • We can also directly allow users to pass pipelineRef which can be directly used if a user doesn't want to use TriggerBinding/TriggerTemplate
  • A user can provide multiple pipeline templates which can be triggered together

Available solution: There is an operator by Kevin https://github.com/bigkevmcd/tekton-polling-operator which polls a repo and create a pipelinerun/taskrun if something is changes. This is independent of TektonBinding and TektonTemplate, it take PipelineRef as input.

cc @vdemeester

found this error : " no matches for kind "SyncRepo" in version "triggers.tekton.dev/v1alpha1""

vdemeester commented 2 years ago

[…] found this error : " no matches for kind "SyncRepo" in version "triggers.tekton.dev/v1alpha1""

@tisaiful31 this is to be expected, what I wrote is just a proposal, that's still being discussed and thought trough.

bigkevmcd commented 2 years ago

@vdemeester I have working code that I'm literally just writing examples and docs for, that simplifies this to just a CR that detects changes in a ref in a git repository and sends a CloudEvent on change...not Tekton specific at all

dibyom commented 2 years ago

@bigkevmcd That sounds like a really nice solution to this

triantium commented 2 years ago

There is a implementation that works in combination with FluxCD https://github.com/jquad-group/pipeline-trigger-operator

They use the Flux Sourcecontroller to track changes.

bigkevmcd commented 2 years ago

@triantium that is a nice idea, it could of course be connected to triggers via flux notifications like this https://bigkevmcd.github.io/tekton/flux/notifications/2020/12/06/flux-tekton.html

tekton-robot commented 2 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten with a justification. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

vdemeester commented 1 year ago

/area roadmap

Syntax3rror404 commented 1 year ago

Are there any news regarding this feature? We also cannot use webhooks because of a corporate proxy :/ This issue is now 2 years old ...