search

Strider-CD / strider

Open Source Continuous Integration & Deployment Server
http://strider-cd.github.io/
4.59k stars 432 forks source link

Feature Request/Proposal - Strider Scheduler Plugin #538

Open bitwit opened 10 years ago

bitwit commented 10 years ago

Hey,

So I've been interested in helping get the command line tasks in order because we'd like to run scheduled test/deploys, but in the bigger picture a scheduler plugin is more in line with the mantra of simplicity that Strider embraces.

I'm happy to start taking a stab at it but wanted to open a discussion first for some guidance since you guys obviously know the anatomy of Strider (and likely CD best practices) better than I do. A couple things:

  1. Using a scheduler plugin would replace the need for creating a cron job. So would some sort of 'setInterval' be the best way to regularly check if there's a scheduled job to run? Is there already an interval running that I could tap into rather than starting my own?
  2. Typically would the scheduler run a job every time it is scheduled to, or check if there is actually a new commit since the last commit tested on that branch? My guess would be that it only runs a new job if there are actually new commits.
  3. Are there any key features or interface styles I should consider or take inspiration from that you would suggest? I've only used OSX Server and toyed with Jenkins prior to implementing Strider.

Thanks!

jaredly commented 10 years ago

Hmmm here are my thoughts

  1. the scheduler plugin should be responsible for managing its own setTimeouts, both a) on startup and b) whenever the settings are changed
  2. make it a setting. I could imagine "run no matter what" being desirable to check dependencies, for example
  3. tbh, we haven't put a whole lot of sweat and tears into designing the interface, so do what looks nice and is easy to use
bitwit commented 10 years ago

The scheduler plugin is well under way! -> https://github.com/bitwit/strider-scheduler

So a few things: (See webapp.js for reference to my comments below)

  1. I'm tapping into routes as my init function and to grab a reference to the context, is this the best place to be doing this or is there an actual init I can use much like the 'basic' plugin type offers?
  2. Currently the scheduler starts a timeout on start up and runs a check on all the projects and their branch plugin statuses every single tick. I think this should only be done on once on init rather than every time, ESPECIALLY if the scheduler plugin is not currently attached to any live project. However, I couldn't find any emits that would allow me to listen for the plugin being added/removed from a project branch. Does this exist? If so, I can make sure that a timeout loop only exists when a branch is using the plugin and also monitor only the subset of projects using the plugin after init.

Cheers

jaredly commented 10 years ago

Hmm yeah I don't think we have events emitted for "added to project" or "removed from project". And we'll also want "config saved for project", I imagine. Do you want to jump in and add those?

for "saved for project" we have https://github.com/Strider-CD/strider/blob/master/routes/index.js#L154 and for "added to/removed from", this should be the place: https://github.com/Strider-CD/strider/blob/master/routes/index.js#L180

bitwit commented 10 years ago

Haha, good news, I have a PR ready for the 'added to/removed from' already and I'll add the other one too. Then I can revise this plugin. Will get back to you in a few days.

jaredly commented 10 years ago

sounds great :)

bitwit commented 10 years ago

So these emitter events are working great and helping the plugin run a tight scheduling process. At this point the plugin would be near ready if it just needed to run jobs on a schedule much like the cron job command can already accomplish.

However, I think adding the commit check capability is a pretty key feature. So the best I've been able to do so far is check the hash in worker.js right AFTER git clone during prepare if it is at the top of the plugin order. Nice, but not ideal given the ordering requirement and overhead of starting a job only to kill it after a clone that takes 'x' long to pull down.

What would be ideal is to run a 'git ls-remote' command from the webapp.js, so I'm wondering how it would be best to go about working with the provider to access the git repo with adequate credentials and retrieve this information.

Thanks.

bitwit commented 9 years ago

Okay, figured this out by reading into the strider-git repo which appears to have just about everything I need. Gonna take a few weeks to flesh it out as I'd like but I think I've got everything I need now.

Cheers

bitwit commented 9 years ago

Finally had the chance to wrap up the first version of scheduler this weekend!

https://github.com/bitwit/strider-scheduler

You can schedule 3 different ways right now:

  1. Set a branch to be polled every minute for new changes
  2. Set an hourly job to run for a branch
  3. Set a daily job to run for a branch, selecting the specific days you want to run. i.e. you can disable Saturdays and Sundays

Strider scheduler is going to need whatever version is cut next after 1.5.0 since it uses the new plugin emitter events.

Also, I think the poller needs more testing with different versions of git. I've only tested this with GitHub so far.

Any feedback on the scheduler plugin is much appreciated. I'll keep testing on my side too and looking for hiccups.

niallo commented 9 years ago

Wow this looks awesome! I'm definitely in favour of a scheduler plugin being included in the default install.

On Mon, Oct 13, 2014 at 3:56 PM, Kyle Newsome notifications@github.com wrote:

Finally had the chance to wrap up the first version of scheduler this weekend!

https://github.com/bitwit/strider-scheduler

You can schedule 3 different ways right now:

  1. Set a branch to be polled every minute for new changes
  2. Set an hourly job to run for a branch
  3. Set a daily job to run for a branch, selecting the specific days you want to run. i.e. you can disable Saturdays and Sundays

Strider scheduler is going to need whatever version is cut next after 1.5.0 since it uses the new plugin emitter events.

Also, I think the poller needs more testing with different versions of git. I've only tested this with GitHub so far.

Any feedback on the scheduler plugin is much appreciated. I'll keep testing on my side too and looking for hiccups.

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/538#issuecomment-58966966.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

logankoester commented 9 years ago

Awesome, definitely a feature I've been wishing for also.