Proposal: Release Process

[mikeal]

Several discussions and threads have made it clear that the existing release policy is not working for a lot of users and developers. At NodeConf Adventure we had a few sessions about this and, with more users than node core developers, we came to very different conclusions than we have in prior threads and TSC meetings.

First, I should lay out the different concerns the release process needs to satisfy.

• We need to take new versions of v8 in a timely manor.
  • We need to do real releases (not just nightlies) that are in use.
  • If we don't do this we fall too far behind the v8 cycle and can't collaborate effectively.
  • There's a growing contingent of users that want to adopt new language features and would like to have access to them on roughly the same schedule Chrome is releasing them.
• We need to release contributions in a timely and usable manor.
  • The vast majority of contributions are not breaking changes but patches and feature additions.
  • The current schedule for breaking native addons means that most contributions are being released in a version that is not usable by the vast majority of users because the timeline for native addons being updated is longer than our major release cycle.
  • We need an incubation phase for these releases to validate there are no breaking changes. The number of breaks we've had in CURRENT leads people to avoid it and actively advocate that people don't run it.
• We need a release line that is extra stable and a way to signal when a breaking release in this line is ready to be adopted. Actual quote: "I won't run a release line that hasn't been in wide use for a year."
• We need LTS releases and an LTS guarantee that is achievable. On a 5 year support timeline this means that we have to support (major releases per year * 5). If we do a six week major cycle that means 43 support lines -- that is not achievable.

One thing that should be clear by now is that users who wish to "adopt new JS language features" cannot rely on native addons. Those two priorities are simply incompatible at this time. We have good ideas about how to improve this in the future but we aren't there yet and v8 is going to break native addons each release for the foreseeable future.

Here's my proposal, laid out by branch.

master branch

Master should release on the same cycle and frequency we have now. Releases should appear in the BETA channel. If a critical bug or regression is found the PR fixing it should be tagged and merged.

incubation period

Master releases appear in the beta channel. If a critical bug is not tagged and merged within the incubation period the last release older than the incubation period is moved to the CURRENT channel.

next branch

Multiple updates to v8 should appear in the next branch. Relatively frequent releases should appear in the CANARY channel with increasing alpha version numbers (i.e. 3.0.0-alpha19). Any breaking changes being considered should also appear in the next branch.

Approximately once a year the next branch should be merged in to master and all the appropriate version numbers kicked over. This is at the discretion of the TSC but should likely happen when there is a good level of confidence in the stability of v8.

LTS branches

When next merges in to master we create a new branch for that major release's LTS line (as we do today).

The STABLE channel contains that latest LTS release of the latest major version under LTS.

Website Mockup

---------------------------------------------
|         |         |        |              |
| STABLE  | CURRENT |  BETA  |    CANARY    |
| 0.10.99 | 2.89.4  | 2.91.2 | 3.0 Alpha 14 |
|         |         |        |              |
---------------------------------------------

User Expectations

Developers who want to adopt new features will follow the CANARY channel. We can do evangelism around major events in the alpha cycle based on features that land from v8 or other major changes that might land. We should expect this section of the community to slowly move away from any dependence on native modules.

Most people involved in regular core development will sit on the BETA channel. Large institutions and popular open source projects should point their test automation at the BETA channel and encourage their users to adopt the CURRENT channel. The CURRENT major lives for about a year, which is about how long it takes for the entire native ecosystem to "catch up."

Larger production users should track STABLE. When upgrading to a new major release of STABLE the expectation is that all modules will work and have ample testing over the year of being in the CURRENT channel.

Current Versions, Branding and the Convergence.

If we adopt this release process we should scrap our current convergence plan.

Features from 0.12 should converge against io.js master. master should remain at 2.x. No breaking changes can be converged in master and should instead be done on next.

All release lines should rename to "node.js" as soon as possible and development should move to nodejs/node as soon as convergence on master is complete.

• STABLE is 0.10.x
• CURRENT and BETA are 2.x
• CANARY is 3.x

I don't know what this means for LTS of node.js 0.12 and io.js 1.x. Maybe we do it because it sends the right message but I don't think that either of these release lines should land in STABLE and we should skip over them because neither will end up going through a year in the CURRENT channel which I believe should be a requirement for this channel.

[ahmadnassri]

I'm not certain whether LTS should or shouldn't be part of the release process considerations? that is to say, there doesn't seem to be any reasoning to limit a "release process" design with identifying LTS candidates.

you could in theory identify any major release as candidate for LTS, regardless of numbering or timing, that could be determined in a separate policy / process.

I can also see there's a working group discussing LTS already, so some more clarity hopefully could surface from that side.

[mikeal]

you could in theory identify any major release as candidate for LTS, regardless of numbering or timing, that could be determined in a separate policy / process.

This is what I had thought we could do a few months back but, unfortunately, it isn't quite practical. Users look for an LTS guarantee in anything you ask them to adopt and we are certainly asking people to adopt CURRENT. If you do releases that you ask people to adopt and then drop support shortly after you don't end up accomplishing what LTS is designed to do: create a feeling of safety around adoption of the platform.

[chrisdickinson]

My initial reaction: I'm broadly in favor of this. We might consider changing the names of STABLE vs. CURRENT since they have the potential to direct audiences to the wrong place (namely, we want enterprise customers to choose "STABLE", we want users to choose "CURRENT.") That's a naming quibble, though.

I'm going to stew on this a bit more (especially the convergence bits), but for now, great work!

[mikeal]

@chrisdickinson I sort of stole the naming convention from FreeBSD for CURRENT and STABLE. It seems to work for them but our community is quite different so we may need to use something else.

Also, keep in mind that module authors also need a timeline for dropping support of old releases in their current development so we need something that conveys an expectation to module authors that they should still be actively supporting and testing against whatever we end up calling the STABLE channel.

[mscdex]

I also think that having both "Current" and "Stable" labels as separate things can be confusing, since it's possible to have a "current" stable version and an old stable version (e.g. v0.10.38 vs v0.10.0 or even v0.8.xx). I'm not sure what name(s) would be better though.

[chrisdickinson]

@mikeal I was (a bit cheekily, perhaps) considering calling the "STABLE" channel the "ENTERPRISE" channel. Plus, this lets us start building up a space-/sf-themed set of channels (io.js, enterprise.js, etc, etc. – just need ones for "beta" and "current") :alien:

[mikeal]

considering calling the "STABLE" channel the "ENTERPRISE"

I'm not opposed to this.

set of channels (io.js, enterprise.js, etc)

Perhaps some time in the future we can call one io.js but, at the moment, calling one of them io.js is deeply confusing :)

[ahmadnassri]

you could in theory identify any major release as candidate for LTS, regardless of numbering or timing, that could be determined in a separate policy / process.

This is what I had thought we could do a few months back but, unfortunately, it isn't quite practical. Users look for an LTS guarantee in anything you ask them to adopt and we are certainly asking people to adopt CURRENT. If you do releases that you ask people to adopt and then drop support shortly after you don't end up accomplishing what LTS is designed to do: create a feeling of safety around adoption of the platform.

I don't disagree with that statement, I guess all I'm saying is: you can create a separate policy for tagging a major release as LTS, regardless of what process you use to issue releases.

e.g. the LTS policy can be as simple as: "Every second major release is an LTS release", so: (major releases per year / 2).

or perhaps tie LTS with yearly quarters: 4 LTS releases per year * 5 years = 20.

[mscdex]

Does anyone have any opinions on having an LTS release schedule like that of Ubuntu's (currently an LTS release every 2 years)?

[Fishrock123]

We need LTS releases and an LTS guarantee that is achievable. On a 5 year support timeline this means that we have to support (major releases per year * 5). If we do a six week major cycle that means 43 support lines -- that is not achievable.

I think the current LTS proposals are widely misunderstood. To be fair, they are also clumsy and confusing.

The previous idea would be to widely message that only some lines will be tagged for LTS.

[mikeal]

I think the current LTS proposals are widely misunderstood. To be fair, they are also clumsy and confusing.

I think this is mostly because we have been writing them in isolation from the actual release process, which we keep changing and then expecting the LTS folks to catch up.

At various points we've externalized LTS needs by saying "the LTS WG will handle that" so we don't have to consider them in the regular release process and effectively treated the LTS WG like a dumping ground for stability concerns. Stability concerns must be a part of the regular process and the LTS WG can find the best way to coordinate the effort required to do STABLE and LTS releases once the regular process, which appreciates the time and effort required to maintain a major release line, shifts the major release cycle.

[subfuzion]

I think we should avoid conflating STABLE with LTS. If we consider the Ubuntu community's definition of LTS, it is enterprise focused and not a feature-based / cutting edge release.

Using @mikeal's website mockup, it would look like this.

 -------------------------------------------
|         |         |        |              |
|   LTS   | CURRENT |  BETA  |    CANARY    |
| 0.10.99 | 2.89.4  | 2.91.2 | 3.0 Alpha 14 |
|         |         |        |              |
 -------------------------------------------

I agree with @mikeal that CURRENT needs to be stable. There is no point in having current releases perceived as beta releases that organizations are afraid to adopt. I like the idea of an incubation period for releases before promoting them to the next channel.

I disagree with the need to indicated a stable line that has been in wide use for a year. LTS provides a strong stability guarantee for those who need it. Just because a release line is a year old doesn't mean it conveys any guarantee about how widely adopted and therefore vetted it was, particularly given the rapid rate of io.js releases.

An organization that wants to hold off on adopting current releases for a period of time as a matter of policy can certainly do so, but labeling an older release as stable is potentially misleading. Especially with io.js' adoption of semantic versioning, the decision whether to adopt current, remain a minor (or major) version behind, or stick with LTS should be enough for an organization.

[rvagg]

(commenting without having read the bulk of the comments yet, sorry)

Can we please remove discussion of LTS from this thread, there is almost zero overlap between the proposal(s) here for LTS and the work that's been happening in the LTS WG (on GitHub exclusively so far). This is a complex topic and requires significant commitment from certain parties that have a stronger interest in LTS than most users (as nice as it sounds to have "stability", it's not the new hotness that everyone really wants to work on, LTS work is going to be a hard, borning, tedious slog that will likely be supported primarily by corporate players with a commercial interest in this area).

So rather than allowing this discussion get into fantasy-land and therefore go nowhere can we please, please, please remove LTS from the scope and focus on other aspects of "release" instead. If you are really interested in LTS (and can realistically contribute) then pipe up in the LTS WG--as much as everyone wants to talk about it, it's been difficult to get people who will do the hard work together thus far but we'll be trying again in the coming weeks.

[subfuzion]

@rvagg, Confining LTS discussion to the LTS WG makes sense (and I completely agree with LTS being tedious -- and likely supported primarily by commercially interested, corporate players and therefore needing separate consideration).

With that being the case, I also understand the need to indicate a "stable" channel as initially presented in @mikeal's mockup. Although I still chafe at the word, there's no doubt in my mind that 0.10.* represents a pretty stable version of Node at this point. I'm not sure at which point I began to think of it as stable, but I suppose it was at some time after it had been been out for a while and gone through a number of updates. So I guess that brings me right back to the original then:

 -------------------------------------------
|         |         |        |              |
| STABLE  | CURRENT |  BETA  |    CANARY    |
| 0.10.99 | 2.89.4  | 2.91.2 | 3.0 Alpha 14 |
|         |         |        |              |
 -------------------------------------------

I guess what I hate about "stable" is that it will be used by so many people as a reason for not using current. Maybe we could call it VENERABLE ;)

[mikeal]

Some notes from a recent conversation with @rvagg on IRC.

• While we figure out this whole canary/beta/current thing we should stay away from discussing the LTS process. For the time being 0.10.x is the blessed LTS and the process for maintaining and releasing it can go on while we figure this out.
• Current, next is a bit stale so if we're talking about persisting it for a while we'll need to come up with a good way to keep it up to date with what lands in master
  • Possibly automate a rebase.
  • Possibly do time based releases to ensure we don't wait too long between.

[subfuzion]

@mikeal, when you say:

• We need to take new versions of v8 in a timely manor.
  • We need to do real releases (not just nightlies) that are in use.

what do you mean by "we need to do real releases"? Are you referring to the v8 releases?

And when you say:

Master releases appear in the beta channel. If a critical bug is not tagged and merged within the incubation period the last release older than the incubation period is moved to the CURRENT channel.

this means that if a critical bug IS tagged and merged within the incubation period then the previous release is NOT moved to the CURRENT channel and the incubation period is extended by the new BETA fix, correct?

Any thoughts on a reasonable incubation period? 10 days? 30 days?

[benjamingr]

• 1000 for this. The way the current release process works is broken and this would go a long way to fix it. I think that time bounds need to be established so stable never falls too far back - we don't want a 0.10 and 0.11 line.

[mikeal]

what do you mean by "we need to do real releases"? Are you referring to the v8 releases?

Real releases of Node. It's hard to build a community round nightlies, having releases you can point people at and get them excited about a feature landing help, which is why I think that Alphas are a good idea.

Any thoughts on a reasonable incubation period? 10 days? 30 days?

The time that was kicked around at Adventure was 2 weeks. I think we can start there and iterate over time as we learn.

[Fishrock123]

The time that was kicked around at Adventure was 2 weeks. I think we can start there and iterate over time as we learn.

I'd like to try starting with one week ala npm. I think it was mostly you kicking around the 2 week thing. :)

[mikeal]

I'd like to try starting with one week ala npm. I think it was mostly you kicking around the 2 week thing. :)

1 week sounds even better.

[aredridel]

I think with the pace of things, 1 week makes a lot of sense.

[Fishrock123]

If we adopt this release process we should scrap our current convergence plan.

Features from 0.12 should converge against io.js master. master should remain at 2.x. No breaking changes can be converged in master and should instead be done on next.

All release lines should rename to "node.js" as soon as possible and development should move to nodejs/node as soon as convergence on master is complete.

@mikeal this is confusing, why?

[mikeal]

@mikeal this is confusing, why?

The current plan calls for us to merge everything for the "next major release" but if we're now saying that we might wait close to a year before a new major we would want to begin putting out new converged node.js releases under the CURRENT channel instead.

[domenic]

A year between new major versions makes no sense. A year between new LTS releases seems fine. Let's be sure to differentiate.

[mikeal]

A year between new major versions makes no sense.

That's about how long it takes for the native module ecosystem to catch up, so it sort of does make sense -- assuming that we have a channel where v8 is adopted on a 6 week cycle and used outside of the major cycle (Alpha releases in the CANARY channel).

[domenic]

Firefox is a good example here. Every 7 versions is a LTS release. But major version bumps every 6 weeks. That's a reasonable model and one we should be following.

[mikeal]

Every 7 versions is a LTS release.

Each one of those 7 releases doesn't break 30% of the websites it tries to render. Those aren't breaking releases -- our problem is that we have major releases with large breaking changes to native modules.

[domenic]

assuming that we have a channel where v8 is adopted on a 6 week cycle and used outside of the major cycle (Alpha releases in the CANARY channel).

Relegating up-to-date, security-bugs-fixed JS VMs to CANARY is not tenable. We should not advertise insecure releases. We should advertise LTS, which gets security work done by the LTS community, and we should advertise current major, which is secure and usable.

our problem is that we have major releases with large breaking changes to native modules.

Yes, that is the definition of major release.

[domenic]

If some native modules only work with LTS, that's fine. That shouldn't slow the pace of major releases.

[mikeal]

@domenic I think you're getting too hung up on version numbers here -- and you're conflating LTS concerns which we previously agreed to remove from this discussion. Can you talk about the users and technical concerns not addressed by this plan?

[domenic]

I don't think I am getting too hung up on version numbers. I think that once you remove LTS concerns there is not much left, so that is why this thread is baffling. This plan seems entirely pointless given a reasonable LTS strategy---so I don't see what users and technical concerns it addresses.

[mikeal]

@domenic the majority of users seem very dissatisfied with the current plan actually, most of them not LTS users.

[domenic]

That's because we don't have a working LTS model yet!

[mikeal]

@domenic I don't think the users we're talking to want LTS releases, they actually want to run current releases where new features and patches land somewhat regularly -- they just don't want things to break all the time.

[domenic]

Users who want native addons not to break, given that according to you native addons take about a year to update, should definitely be running LTS releases with a one-year stability guarantee.

If you think LTS releases should backport feature work in addition to patch, that's a fine discussion to have in the LTS working group, IMO.

[mikeal]

@domenic if the vast majority of users are stuck on LTS that actually isn't good for the project. We need people using newer features on a faster timeline than that for a number of reasons I already outlined. We also need people who want new JS features from v8 adopting those features on a better timeline. We need to do both of those things and those users should represent the majority of users (although possibly not the majority of deployments).

[domenic]

if the vast majority of users are stuck on LTS that actually isn't good for the project

Agreed. I don't see how this has anything to do with what I've been saying, though.

[domenic]

In the end this comes down to security. You have two sources of secure JS VMs:

• The one maintained by the LTS working group
• The one maintained by the V8 team

Everything follows from there.

[mikeal]

@domenic breaking changes landing in CURRENT (both because of lack of incubation and because of native addon breaks) are the biggest barrier to people using CURRENT.

You have two sources of secure JS VMs

How long can we expect major version line of v8 to be maintained by Google?

[domenic]

How long can we expect major version line of v8 to be maintained by Google?

6 weeks.

[mikeal]

6 weeks

It's at least 12 weeks -- they support the currently releases version as well as the upcoming version (this is assuming we are taking the active CANARY version).

But, I've seen them continue some support and patches of older lines longer than that in the past but only for the last major release, the timeline for major releases is totally unclear to me though.

[domenic]

No, upcoming releases are not supported, and neither are past ones. Maybe there have been one-off exceptions on request, but that is done by engineers in their spare (non-Google) time.

The timeline for major releases is every 6 weeks (synced with Chrome).

[mikeal]

@domenic ok, so what you're saying is that if we were to sit on the same version of v8 in the CURRENT channel for a year we'd have to maintain that v8 branch ourselves just like we have to do for LTS? Effectively, CURRENT would have a v8 under LTS while the alpha releases would be pushing supported releases of v8 on the 6 week timeline of Chrome.

[domenic]

Yes. That would take care of the security aspect.

Separately, I think it would be terrible to relegate people who want to use (or write libraries depending on) new language features to a channel designated "alpha." Or people who want a more performant VM, or people who want to use new V8 C++ APIs.

Maybe this is just a branding thing though. Maybe the one that uses a current JS VM would be named "current" or something, whereas the one using a LTS VM would be named "LTS with features" or something.

[mikeal]

Maybe this is just a branding thing though.

I think that it is. On reflection CANARY may not be the best name. The important part is that the channel needs to convey that "You will get great new v8 stuff, you will need to remove your dependence on native modules if you depend on this channel."

[domenic]

+1.

With the caveat that, in the medium-term, I still believe we can do better with native modules and shorten the feedback loop, as discussed elsewhere. (I.e., have releases ready for testing 6-12 weeks in advance, have automated testing and perhaps even automated PRs, etc. so that in theory by the time Chrome rolls some large portion of the native module ecosystem is ready. Plus, V8's list of major upcoming breaking changes is settling down, supposedly, but I know that's not exactly a promise that has much behind it.)

[mikeal]

@domenic so, this proposal is based on three things that are currently the reality but if either change we should consider changing the policy.

• v8 releases, which occur every 6 weeks, will break most native addons.
• most native addons break and require some kind of modification and re-publish when v8 breaks.
• the rough timeline for most of the native ecosystem to catch up is a year.

If any of those things change we should consider a more aggressive major release cycle, and I do think some of them will eventually change.

[mikeal]

To try and reign this in, a few things I think that we agree on:

• master should be released frequently.
• master releases should have an incubation cycle to ensure we don't accidentally cause breakage.
• next should adopt new v8 versions on a 6 week cycle.
• next should be continuously kept up to date with improvements to master

In order to move this forward, let's try to find more things that we can agree on. In order to do that lets stay away from a few things:

• version numbers
• how we market channels and what those channels are called
• releases of branches that are not master or next

We'll figure those out after we agree on some other things.

[mikeal]

Here's a few things I'd like to find agreement on.

• next should not be merged with master on shorter than a one year timeline while it breaks virtually all native addons.
• next should have regular releases, not just nightlies as we've been doing.

[domenic]

Here is my proposal based on the TSC call discussion.

Current setup:

• LTS: used by enterprises. No new features, only bugfixes. Requires us to maintain a V8 fork.
• "master": used by everyone else. Major releases every 6 weeks (in theory). Has problems that spurred this thread.

Proposed setup, using meaningless codenames so as to avoid marketing debates:

• LTS: unchanged (trying not to discuss LTS here as much as possible)
• "A"
  • Similar to current master.
  • Major releases 6-weekly
  • Version number increments with every release, like normal.
  • Has regular "release candidate" or "beta" versions that people test to help us avoid bad things (like breaking changes during what we thought was a patch or minor).
  • Used by people who:
  • Are OK with breaking changes, in particular with regard to breaking their native add-ons that do not keep up to the 6-week pace. (Most other breaking changes have been minor.)
  • Acknowledge that sometimes we might accidentally release a patch/minor with a breaking change, even given the beta safety net. (We'll of course fix this ASAP.)
  • Want new V8 features/performance/APIs.
• "B"
  • New.
  • Major releases yearly.
  • Each major release picks a version number corresponding to the "A" version that it branches off of.
  • Requires us to maintain a second V8 fork.
  • Used by people who:
  • Want to experience breaking changes approximately once per year
  • Are never OK with seeing accidental breaking changes
  • Want to get new features, in addition to just LTS-style bugfixes
  • Are willing to forgoe V8 improvements

So people who heavily use native add-ons will use "B" releases, whereas "A" releases are otherwise the best option.

Library authors will then choose which versions they support. E.g. jsdom has a 3.x branch that supports LTS, and a (say) 6.x branch that supports the 2015 "B" release, and then master jsdom will work with "A". But more conservative libraries may want to support LTS onward, or 2015 "B" onward, for an indefinite amount of time.

---

Here's a few things I'd like to find agreement on.

• next should not be merged with master on longer than a one year timeline while it breaks virtually all native addons.

I think this is a process issue for the collaborators to decide, to be honest. So far the spirit for the collaborators seems to prefer master being up to date. So maybe I'd phrase this as "A should not be merged into B on longer than..." and then the collaborators decide where master is A or B.

• next should have regular releases, not just nightlies as we've been doing.

Definitely. In my above this is the release candidate or beta versions of "A".