Work together to generalize jenkins setup?

[patcon]

I've known about this awesome project for awhile, but I'm just now digging in to see how it works.

I've currently got a similar project that I'm planning to revamp, and was wondering whether y'all would be interested in at least entertaining the idea of collaborating.

Here's our project: https://github.com/myplanetdigital/jenkins-inception

Obviously lots different, but I'm wondering if you'd be interested in discussing the sharing of a common "core". Our current setup uses a few rake tasks talking to various services (rackspace, dyndns, github), and delegating the spin-up to knife-solo, using librarian-chef to pull in our cookbooks. I was considering a second iteration to clean it up a bit (who am I kidding -- a lot), and thinking it would likely involve leveraging vagrant providers more fully (rather than knife-solo and using thor to wrap the project up as a gem (much like you've done).

Obviously the two of us have got some setup specific to how we operate (Drupal and build-pipelines), and you've got your own, but maybe we could somehow draw that out? Maybe we could create a way to pull in a few extra cookbooks via config and have them stand up the last mile of config? Or perhaps it's too much effort, and that's ok :) Just wanted to reach out

[mkocher]

I've talked about this with @ohrite and @kmayer, I think we're all on board with experimenting with vagrant providers. Ciborg 2.0 predates vagrant providers by about two months, otherwise we likely would have gone that way from the start.

[patcon]

Right on.

The one downside of current Vagrant providers that I've encountered (which is perhaps only inconvenient for how we'd like to operate) is that it's not very simple to make it so that any given team member can run vagrant provision -- Vagrant retrieves the ssh_config info from the cloud provider using api credentials on every command, rather than caching just the ip/port somewhere locally. Basically, we'd like to allow team members (in our case) to update the jenkins env without needing tons of ec2 credentials floating around.

I'd been thinking on how to work around it with the help of this plugin: https://github.com/tknerr/vagrant-managed-servers

Is the credential-centric approach of most vagrant provider plugins inconvenient for you as well? Or should I backburnerize that train of thought for the purposes of collaboration?

[mkocher]

Thinking a little more about this, have you tried out ciborg? There's very little rails specific code, one of the goals of 2.0 was making changing frameworks a matter of changing the run list and changing the ci_build.sh. The first big success for 2.0 was when @pivotal started our first django project and had ci going for it in ten extra minutes.

Build pipelining is something we've got on the horizon. We've been using build-flow on Cloud Foundry, and aren't terribly happy with it.

[patcon]

Re: framework switch. Nice! OK, I think we have similar enough goals. I'll try to fork it and see what would make it work for us :)

I've been talking with some of the opscode folks about using jenkins-job-builder to manage jobs, and have chef set it up: https://github.com/openstack-infra/jenkins-job-builder

Opscode is also interested in working on a jenkins-pipeline cookbook that extends the jenkins cookbook and allows pipelines to be managed via chef LWRP. I'm of the mind that the build pipeline (and by association, all jenkins builds) are best managed at the chef level, but I could be convinced otherwise. The downside is that managing them with chef kinda precludes creating build jobs via the command line, which also seems empowering to teams. So I'm torn.

Re: build-flow. build-pipeline plugin is working great for us. Don't have insight on build-flow.

Action item for me: setting up and testing ciborg

[ohrite]

A while back I decided to try out an idea I had when @mkocher and I were knee-deep in rewrites of soloist and what would eventually become ciborg.

https://github.com/minifast/manic_baker is essentially ciborg, but generalized and aimed at Joyent. It's not really complete, but it's fully capable of bootstrapping and chefing an instance.

The eventual goal is to snapshot a machine instance and enable the parbaking workflow that Netflix uses. Is this of interest?

On Monday, July 15, 2013, Patrick Connolly wrote:

Re: framework switch. Nice! OK, I think we have similar enough goals. I'll try to fork it and see what would make it work for us :)

I've been talking with some of the opscode folks about using jenkins-job-builder to manage jobs, and have chef set it up: https://github.com/openstack-infra/jenkins-job-builder

Opscode is also interested in working on a jenkins-pipeline cookbook that extends the jenkins cookbook and allows pipelines to be managed via chef LWRP. I'm of the mind that the build pipeline (and by association, all jenkins builds) are best managed at the chef level, but I could be convinced otherwise. The downside is that managing them with chef kinda precludes creating build jobs via the command line, which also seems empowering to teams. So I'm torn.

Re: build-flow. build-pipeline plugin is working great for us. Don't have insight on build-flow.

Action item for me: setting up and testing ciborg

— Reply to this email directly or view it on GitHubhttps://github.com/pivotal/ciborg/issues/5#issuecomment-20973265 .

Doc Ritezel +1-415-601-5766

[patcon]

I hate to proclaim my ignorance, but I'm unsure what Netflix "parbake" workflow entails, and so the README is a little unclear? Linky linky? :)

EDIT: Is this good background? http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html