SCM triggering support

[melezhik]

I'd be good to have sparky builds get triggered by changes in SCM:

sparky.yaml:

  scm:
   url: https://github.com/melezhik/rakudist-teddy-bear.git
   branch: main

[thibaultduponchelle]

It looks like a polling approach right ? What about webhooks ?

[melezhik]

Yeah, like an alternative. What do you think will be more useful? Polling or calling sparky API through web hooks? If talk about web hooks, what external system integration do you think of? Travis/Gitub Actions? maybe you have more practical context / use cases on that?

[thibaultduponchelle]

I think polling feature is a nice start. But at some points webhooks will be a must have :) I have a lot of experience with bitbucket/jenkins that offers a scanning process (discover bitbucket project and repo and branches on demand or croned). Branch and repo are configurable on pattern (regex include or exclude). But at the end the usual way to continuous build is using webhook feature. The endpoint is the same for all repo using the same jenkins master (here it would be sparkyd) and repo/branch is therefore probably POSTed but I do not checked this point.

Webhooks are also available on github/gitlab.

I don’t understand your point about Travis ci or github actions that are for me out of picture but can eventually ping the webhook also of course but it is a bit hacky.

Then tldr polling seems ok to start :)

[melezhik]

The reason I am clarifying polling VS web hooks, is that I'd like Sparky to be as close as possible to approached people might follow in their projects ... From technical point of view both polling and web hooks probably has the same level of implementation efforts ...

So, from your point of view, if you used a Sparky to build projects from SCM, what would use, web hooks?

[melezhik]

I don’t understand your point about Travis ci or github actions that are for me out of picture

Sorry for confusion, I just for some reasons thought that web hooks are implemented though github action, but you're right it's not the case ...

[melezhik]

The endpoint is the same for all repo using the same jenkins master (here it would be sparkyd) and repo/branch is therefore probably POSTed but I do not checked this point.

yeah, I see. Some of the work is already done - https://github.com/melezhik/sparky#trigger-jobs-from-http-api

[thibaultduponchelle]

Ah yes, then you already have webhooks :)

And yes the answer to your question is that I would personally use webhook. (But the polling is easier to put in place from a beginner point of view)

[thibaultduponchelle]

Just for clarification, despite usually being the same endpoints, webhooks are usually set at repo level (means that having a different webhook per repo should work). Not sure if we can set it at bitbucket project level (I can check, but I’m almost sure we can’t) or github organization level (dunno I don’t have one 😁) or similar other scm solutions. If it’s not clear : a project for bitbucket means a set of repo like a github organization.

[melezhik]

I see that. Anyway Sparky post request differentiates on a project level as well.

On Fri, Dec 25, 2020, 2:10 PM thibaultduponchelle notifications@github.com wrote:

Just for clarification, despite usually being the same endpoints, webhooks are usually set at repo level (means that having a different webhook per repo will work). Not sure if we can set it at bitbucket project level (I can check, but I’m almost sure we can’t) or github organization level (dunno I don’t have one 😁) or similar other scm solutions. If it’s not clear : a project for bitbucket means a set of repo like a github organization.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/melezhik/sparky/issues/3#issuecomment-751286906, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHRHSLTQPPJLMNXVH6T5J3SWTWUDANCNFSM4VJI7OIA .

[melezhik]

I decided to implement polling first, I'd appreciate if someone test it out ...

[thibaultduponchelle]

Works like a charm \o/

[thibaultduponchelle]

And webhook is also working !!

I used one of my github repo and your rakudist sparky for testing :

And it triggered well sparky jobs :

Not bad ! :+1: :laughing:

[melezhik]

Hi! Thank you for testing. Yeah web hook should trigger a sparky build like you see. The problem with it - it does not see any post data passed through a trigger, b/c - https://github.com/melezhik/sparky/blob/master/bin/sparky-web.raku#L15 like you see it does not pass any post data back to Sparky build. I'd create a respected ticket on web hooks support latter. Now I am closing this one. Thanks for testing 😄

[melezhik]

btw when you trigger build remotely using POST request it runs with default sparrowdo settings ( b/c there is no sparrowdo settings set for https://rakudist.raku.org/sparky/project/RakuPlay-1 project ), e.g:

with sudo, and with bootstrap and on localhost. Sudo is not enabled for a user that runs Sparky on the rakudist host, so I have to kill hanging jobs manually:

$ ps uax| grep sudo
root      70042  0.0  0.0  62232  4184 pts/2    S+   16:18   0:00 sudo --login sh /home/rakudist/projects/RakuDist/sparky/RakuPlay-1/.sparrowdo/bootstrap.sh
rakudist  70106  0.0  0.0  13148  1052 pts/0    S+   16:27   0:00 grep sudo
$ kill 70042
rakudist@brezeleisen:~$ ps uax| grep sudo
rakudist  70108  0.0  0.0  13148  1076 pts/0    S+   16:27   0:00 grep sudo

Long story short projects RakuPlay-1,2,3 are meant to be run using file triggering protocol - https://github.com/melezhik/sparky#file-triggering-protocol-ftp , using https://rakudist.raku.org/play/ app client - https://github.com/melezhik/RakuPlay/blob/main/lib/RakuPlay.rakumod#L55-L59 ( this is where all sparrowdo settings are set in trigger file ) , but not directly though a POST request