Picking the right CI tooling

[andrewlow]

Today Node is built using Jenkins.

Given the scale of what we want to do with Node, should we take this opportunity to consider a new CI tool?

If yes, we should generate a list of candidates and discuss.

[mikeal]

Prediction: this thread will end in people wanting to build a new CI tool in Node and bike-shedding features :)

[andrewlow]

Oh no! Not my intention at all. Building a new tool is a fine idea, but that's not a good choice as it'd have no prior proof of being useful. If you want to build a CI tool in Node - go build one, but not here please.

I have quite a bit of experience with Jenkins, and my personal opinion is once you grow past a couple of dozen jobs, it becomes unwieldy to manage.

There may be other CI tools which will deal with the scope of what we want to do better. One that springs to mind is http://buildbot.net/ - so I'll suggest that as one alternative.

[mikeal]

A good thing to remember as well is that, as important as picking a good tool, is picking a tool that enough people are comfortable using who are stepping up to do the work. In that case, along with suggestions people may want to put down their experience, as you have, with different build systems and also how much work they are putting in to this particular project.

[rmg]

The more I use Jenkins the less comfortable I am with it. I haven't used buildbot before, but the fact that Chromium uses it is encouraging (http://build.chromium.org/p/chromium/waterfall).

Based on the other thread, @dshaw @ingsings @pquerna and @rvagg might already be coordinating resources, which may or may not mean a CI tool has already been chosen by those doing the work.

[andrewlow]

Mozilla uses buildbot https://wiki.mozilla.org/Buildbot as well.

I'm not sure if the resource coordination thread https://github.com/node-forward/build/issues/1 has locked anyone into tooling just yet.

I do agree with @mikeal that familiarity with the tooling on the part of those doing the work will also play a role here. I'm willing to help out to get the CI tooling setup and roll together some builds. Ideally the core technical team needs to also feel comfortable with the tooling, not just those that care about getting builds out the door.

[rvagg]

I'll shave some yaks and come up with a more firm opinion about Buildbot vs Jenkins. What I do know is that we have lashings of JVM experience laying around that would likely be useful, plus the existing Jenkins infra that exists and the expertise around that.

[rvagg]

I'd love to support drone.io but their tooling is Docker specific, however they are looking to deal with this because one of their goals is to "replace Jenkins": https://github.com/drone/drone/issues/135

Sadly not ready yet.

[rvagg]

CruiseControl and maybe Apache Continuum are the only other serious candidates I'm aware of; I find this landscape depressing and ready for Disruption^:tm:.

[bnoordhuis]

Until 0.4 or 0.6, node.js used buildbot in an ad-hoc fashion where individuals donated machines. It was okay, I liked it. The benchmark page, for example, made it real easy to spot performance regressions.

We eventually stopped using it but that wasn't because of any fault in buildbot. I don't remember why we settled on Jenkins but @tjfontaine probably does.

[bnoordhuis]

Apple is in the process of setting up a new CI cluster for LLVM and they're switching from buildbot to Jenkins (link).

I infer from the email thread that the reason for the switch is that Jenkins has more out of the box functionality. Sounds plausible but undoubtedly also depends on your use case.

[andrewlow]

The current https://github.com/node-forward/build/blob/master/README.md calls out Buildbot as the CI tooling.

This is an appropriate issue to comment on concerns with this initial selection.

[rvagg]

Indeed, and let me fill in a bit more detail here. The current thinking (mostly mine) is that there are a few sensible choices here:

1. Jenkins and all that it involves (bloat, yelling at the JVM, etc.)
2. Buildbot and its programatic configuration and messy frontend (and yelling at Python)
3. Some combination of Buildbot and a custom Node.js application because Buildbot is relatively extensible and it's practical to lean on it for what it's good at and replace what it's not good at
4. Something new entirely

As it stands, the README refers to something like # 3, however I'm now leaning more strongly towards #4 for the following reasons:

1. The CI world is in a terrible state at the moment, Jenkins is still "best of breed" but it's pretty terrible and is in serious need of disruption
2. Buildbot is an attempt to do something new but it's still very hacky and requires a lot of programatic configuration so there would be a significant amount of investment in Python code to make this all work and you still end up with a compromised interface that isn't quite designed for what we want but can be sort of made to fit (the Chromium team have done a fairly decent job of customising it for their needs but it's no small effort).
3. There are other new breed tools but they are far from ready. One is drone.io but they are stuck in Docker/container-land and even though they have a goal of "replacing Jenkins" they are mostly distracted with their CI aaS offering which is all about Linux. Their Open Source efforts are worth keeping an eye on but there is basically no significant movement towards a multi-platform solution.
4. Our needs are relatively straight forward and the complication comes in trying to adapt an existing tool to our needs
5. Instead of investing significant amount of time and effort in the JVM, Python or some other platform that doesn't quite suit our needs that same effort could be put towards building something customised to our needs and we could be investing directly in Node.js while doing so. Perhaps we could come out at the end of it with a more serious contender in the CI space.

So that's what I'm exploring right now in parallel with trying to pull together the platforms we need. The basic needs are simple enough that it ought to not be too much of a drain on time. And, of course, this will be open source, although obviously sponsored by NodeSource since there's at least 3 of us interested in working towards this at the moment.

[othiym23]

I think it might be sensible to consider yet another option, which is to bootstrap on the back of either Jenkins (bleah) or Buildbot (less bleah) and do piecewise replacement of the pieces we don't like until we have something we do like. This seems like one of those things where having absolute trust in the toolchain is imperative, and the effort to bootstrap an entirely new solution that satisfies all the criteria currently outlined in this repo's README seems pretty tremendous.

If the plan is to start with something and move towards something a custom solution over time, that means that maybe everything doesn't have to be completely nailed down and perfect to start, and start replacing bits with custom solutions from the rough edges inwards. The goal should be high-quality, repeatable builds first and foremost, and everything else later.

[rvagg]

@othiym23 that was plan # 3 actually, use Buildbot for the build coordination and front it with something better, however, there's the build coordination is not actually the tricky bit, in our case in particular.

[othiym23]

I see it as different – with one, the goal is to have buildbot remain the core thing and give it a prettier / nodier front end; with the other, the goal is to eventually replace buildbot completely, and to not overinvest in making the non-buildbot pieces play too nicely with buildbot given the certainty that eventually it will go away completely.

[andrewlow]

Option # 3 is appealing. It let's us collectively dislike python while giving us a CI system that doesn't require a huge initial investment to get going. We can start by improving the front end, but I can see that over time becoming an effort to replace parts.

Simply extending buildbot to allow for logic written today in python to be implemented in JS (node) would be sweet.

I guess my view is a blend of both @othiym23 and @rvagg - we start with something very basic, but leave the door open to changing everything.

[rmg]

Disclaimer: Of all the yaks in the industry, build systems is my favourite to shave. Years ago, before node and at a previous company, I did option 4. It was written in Python, supported Linux, Windows, and Mac build machines, and had no coordinator (the build agents were self organizing from a shared queue because the cluster was small enough). I've wanted to re-create a similar system in Node for a while because it's basically all I/O shuffling without actually doing anything computationally intensive/interesting.

What I found to be the hardest was picking the right level of abstraction for your build instructions and the output. Running git fetch && git clean && git checkout && make test-simple is easy enough you could just install something like saltstack on all the build machines and just run the commands directly. But that would stop being adequate almost immediately, and the evolution path from that starting point is painful. Starting from buildbot will likely result in a system that feels a lot like buildbot, unless you abstract it to the point where you aren't really building on buildbot anyway. I attribute my dislike of Jenkins to the same effect when I say it was "built for Java in a time before git".

Personally I'm in favour of option 4 because I agree whole-heartedly with @rvagg's observation that cross-platform CI is an inadequately solved problem and I'd really like to see it solved. That said, I don't think I have time to build it myself, so I'd be willing to help with option 3 where/when I can.

[andrewlow]

It did take us nearly a month to complete the cycle https://github.com/node-forward/build/issues/4#issuecomment-51647996 /cc @mikeal

[dshaw]

Haha :8ball:

[retrohacker]

Throwing in my .02, not that it is of value.

/* begin context */

I'm somewhat familiar with Jenkins, and it is indeed garbage. The default toolchain is limiting, and falling back to shell scripts to get things done is another pandora's box of nightmares. It can be extended though.

I have zero experience with buildbot but, if the knowledge in this thread is accurate, it is extremely extensible. That is valuable for our use case.

/* end context */

If we focus on getting a MVP out first, it shows that we can do this and it gets the ball rolling. Even if that MVP is a PITA, it will bring us all together in our mutual hatred for the current solution. This mutual hatred will lead to iteration after iteration resulting in @othiym23's scenario of building buildbot of the solution entirely.

If we start with buildbot (in scenario 3) we end up with a custom solution (scenario 4 in which the oracle @mikeal foretold) and everyone wins.

[Qard]

In any other situation I would be very much against reinventing the wheel. But I despise current CI tools with a burning passion. I'd love to help with option 4 in any way I can! Something composed of small, unixy tools would be super cool.

[z0mt3c]

Didn't find the repo yet. Where did we get stuck? :whale:

[rvagg]

To be honest, we're still at such an early stage that we don't have much to show for ourselves! All current energy is simply getting to all-green so we can start wiring things up to be more automatic and flexible: http://jenkins-node-forward.nodesource.com/

Jenkins is the tool to get us started, and the platforms we're testing are very limited, so the trajectory of progress will be about expanding the test platforms as well as building our own tooling in parallel and that's when things will get interesting! I'll be putting out a call to rally everyone who's interested in collaborating when we're ready for that, I'd love to build a team of passionate people that can contribute to something that's distinct from Node core yet vital to it.

[henrytseng]

Hmmm, building one seems like an interesting project. Would this be a good place to list some features? Or a README.md?

[bnoordhuis]

Is there still anything actionable here? If not, I move to close.

I know @piscisaureus has opinions on how to make interaction with the CI more effective and I personally would like to see us start using something like https://github.com/graydon/bors but perhaps that's something for a new issue.

[objarni]

Pardon my ignorance, but what about Travis CI...? I thought node.js was an open source project?

[bnoordhuis]

@objarni Its matrix of architecture/platform/distro/etc. is not good enough for our purposes. It's also very flaky for anything that's timing sensitive or I/O-heavy. Libuv used it for a while and it was terrible.

I'm going to close the issue, I think it ran its course. Last real action was in November.

[objarni]

OK, thank you. (out of curiosity - what way did you go...?)

[Fishrock123]

Jenkins. (See https://jenkins-iojs.nodesource.com/)

That being said Jenkins is a consistent pain and I definitely wouldn't mind making a Node CI some day.

[cjihrig]

@Fishrock123 it's still in the very early stages, but @lloydbenson has started working on this - https://github.com/fishin/ficion.

[Fishrock123]

Oooh, sweet, I'll check out the code later. :D