search

nodejs / node

Node.js JavaScript runtime ✨🐢🚀✨
https://nodejs.org
Other
106.59k stars 29.06k forks source link

Release proposal: v3.0.0 #1807

Closed rvagg closed 9 years ago

rvagg commented 9 years ago

Currently this is a tracking PR for a 3.0.0 release, some discussion continues on from #1805. This proposal doesn't have a timeframe yet, there are too many unknowns at this stage but we may be able to proceed fairly quickly if things come together nicely.

To be honest I'm not sure exactly what the semver-major here is except for the V8 upgrade and I don't know the details about what's breaking in the C++ API there, perhaps @kkoopa, @bnoordhuis or @trevnorris could fill us in on the details of the breakage for the CHANGELOG? Is this the one where we get all the Maybe* API drama that's going to hurt for everyone?

V8 on the next branch is currently at 4.3.61.21, the CHANGELOG for that is at https://chromium.googlesource.com/v8/v8/+/4.3.61/ChangeLog

The only things on the JS side from scanning the changelog that may be relevant are:

@domenic could I put it on your TODO list to give us a summary of what's changed on the JavaScript end, if anything?

One outstanding item of concern is that I got multiple failures of test-debug-port-from-cmdline.js on OSX, caused by a hanging background process listening on the debug port--I don't know if it was this test on a previous run that caused that hanging process or not, however. If this keeps on showing up then it'll have to hold up a release until we find a fix.

Current log of commits is as follows, but I expect us to get a v2.1.1 out so this will change.

rvagg commented 9 years ago

First release candidate for testing: https://iojs.org/download/next-nightly/v2.1.1-next-nightly2015052770716fdd93/ v3.0.0-RC1 perhaps?

jbergstroem commented 9 years ago

Next seems to fail consistently on an arm7: https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/nodes=armv7-wheezy/714/tapTestReport/test.tap-180/ -- might be a pr/issue for it somewhere. If that's the case, sorry for not searching properly.

rvagg commented 9 years ago

not consistent, but probably about as consistent as the OSX one, see https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/713/ for a successful one for example

trevnorris commented 9 years ago

@rvagg I'm working on finishing the Uint8Array implementation of Buffer behind a flag. Though there are questions I have for the TC meeting tomorrow before it can be finished. This is something I would like to be released on v3.0, if possible.

If it is, I'll give you a full list of any API changes that come with it. Though it will be behind a flag, so one one should immediately feel it.

kkoopa commented 9 years ago

Is this the one where we get all the Maybe* API drama that's going to hurt for everyone?

Yes, every single native addon ever written will need extensive rewriting and it cannot really be automated in a sensible way. Every npm package depending on a native addon will also need updating to pull in the new versions from their upstreams. In total, 64 000 packages may need updating.

This is way worse than those crummy Isolates.

rvagg commented 9 years ago

@kkoopa what's the status of NAN for this? Is it having to wait for 2.0 to make this work? Got an ETA on that (sorry, I've been following but a little detached atm)

kkoopa commented 9 years ago

Definitely 2.0, there is not a snowball's chance in hell of NAN 1.x working on io.js 3.0. nan/next builds against iojs/next. What is missing is mostly documentation and a conversion script. The sooner someone writes this, the sooner it can be released.

imyller commented 9 years ago

My humble suggestion:

Maybe it is time to consider adding and maintaining a Foreign Function Interface (FFI) in the core? Something like the node-ffi provides. Not as a replacement for current addon system, but as a less volatile alternative.

That wouldn't help the immediate package breakage 3.x.x brings, but would steer addon developers who do not need the full V8 native access to implement their own bindings in pure JS in the future.

kkoopa commented 9 years ago

There already is.

imyller commented 9 years ago

@kkoopa Thanks, missed the FFI PR. Hope it goes through. Soon.

kkoopa commented 9 years ago

Simply adding FFI still won't solve anything for many years to come, as people want their addons running on many versions of Node.

imyller commented 9 years ago

I just hope this doesn't turn in to Python 2 vs 3 for Node if the breakage too widespread. 64,000 packages @kkoopa mentioned is a huge chunk of the ecosystem.

kkoopa commented 9 years ago

The problem with Python 2 vs Python 3 was that they kept on supporting Python 2 and even adding new features to it. They should have cut it off completely and let it die quickly and Python 2 would have died off within a year. Python 2 is strictly inferior to Python 3 and it is a shame that it is still in use.

imyller commented 9 years ago

Agreed. My :+1: goes for the incredible and fast progress io.js makes. An outstanding team behind it.

At the same time I'm also bit worried about the quickly progressing fragmentation of packages only supporting 1.x.x, then 2.x.x and soon 3.x.x. Some packages catch up, but at every step the ecosystem looses some permanently.

If important enough people outside the core start making choices like "oh, we are going to stick with 2.x.x because 3.x.x migration has been marginal" then it spells Python-like trouble for Node.

kkoopa commented 9 years ago

In that case they will eventually get forked and updated by someone else with enough desire to support the future. Unlike the Python transition, by using NAN this will still be backwards compatible once updated, so the only reason for not updating is lack of motivation.

Also, while I don't have any figures, I'd imagine that the number of people using io.js 3 will be significantly less than those using any previous release since Node 0.8. For them, there is no imminent need to update every package under the sun the instant a new version is released. It will eventually happen in a gradual fashion as pressure builds up from io.js 3 users.

Also, io.js 2 was a bit of an anomaly in the sense that it was already known that all of this breakage would happen with the release of 3.0 a couple of weeks later, but that's what you get if you insist on releasing a new (major) version every time a dependency is updated every six weeks. There will be more breaking changes in another six weeks when io.js 4.0 is released with V8 4.4. I've tried to make sure all of this is accounted for already with the release of io.js 3.0 and NAN 2.0, to prevent such avoidable breakage, but unless packages take this into account when updating for io.js 3.0, some of them will have more fixing to do for 4.0.

If this release-a-new-major-every-six-weeks-thing is going to continue, it would be advisable to document what is known to break in future updates.

Fishrock123 commented 9 years ago

At the same time I'm also bit worried about the quickly progressing fragmentation of packages only supporting 1.x.x, then 2.x.x and soon 3.x.x. Some packages catch up, but at every step the ecosystem looses some permanently.

It may be worthwhile looking at our plans for LTS: https://github.com/nodejs/LTS/issues

Most modules should probably turn to future LTS for what versions(s) they support, except for modules that want to stay ontop of master. It's likely that the 2.x io.js line won't have one and will be recommended not to have one.

Fishrock123 commented 9 years ago

FWIW I also think it is quite important to get the new Buffer impl (flagged or not) into 3.0.0

mikeal commented 9 years ago

Where does that 64K packages metric come from? Is that the deep dependency analysis of all nan dependents? nan has less than 500 direct dependents so only that many people would potentially need to change their own code, the rest of those 64K packages (assuming this metric is correct) will only need to update the versions of their deps and deps of deps and so on.

domenic commented 9 years ago

Why hold up on Buffer? Especially since it's flagged, we should be fine introducing in 3.1. Seems like a pretty clear "don't hold the train" scenario.

mikeal commented 9 years ago

Just putting this out there, is there a compelling reason to take the new v8 this time around knowing that we're going to have another big round of breakage for the next one?

Fishrock123 commented 9 years ago

we should be fine introducing in 3.1.

It touches a lot of the codebase, I talked to @trevnorris about this and he wasn't particularly comfortable with doing it that way.

domenic commented 9 years ago

Ah ok, I figured since it was behind a flag the impact would be minimal, but I can believe maybe not.

trevnorris commented 9 years ago

@mikeal

is there a compelling reason to take the new v8 this time around knowing that we're going to have another big round of breakage for the next one?

You mean update to v4.3? If we want to get the new Buffer implementation into the wild then absolutely. We can't until at least that version.

@domenic

Why hold up on Buffer?

I'm trying my best to make sure that doesn't happen. The patch is almost ready for review. Reason is because I'm super (possibly overly) cautious about changes to Buffer. Since so much other code relies on it, any hiccup has a strong ripple effects. And even though it's behind a flag, I've had to conditionally change logic for almost every Buffer method. That's the part that makes me nervous.

But if while under review other maintainers say it's fine for release in a minor then I won't prevent that from happening.

mikeal commented 9 years ago

You mean update to v4.3? If we want to get the new Buffer implementation into the wild then absolutely. We can't until at least that version.

We could continue to do that through RC releases of the next branch for now though right?

The thing is, if this breaks the whole native ecosystem this much it won't end up getting much more use than the RC releases get anyway.

kkoopa commented 9 years ago

V8 4.2 was the update that should not have been taken, especially since it was late already. Yes, the 64000 packages is from a walk of the dependency tree, it's an upper bound on packages that may need changing something (code or dependencies does not make a difference).

trevnorris commented 9 years ago

@mikeal If V8 isn't being updated then there's no need for a v3.0 release until we upgrade to v4.4, correct?

domenic commented 9 years ago

I want 4.3 badly to fix many vm bugs #1773. 3.0 now. jsdom wants badly.

kkoopa commented 9 years ago

Not much wrong with 4.3, but if 3.0 is to be released, it should be in a state where it essentially works with V8 4.4 too, so as to avoid doing all of this again in six weeks. We all already know exactly what is in V8 4.4 and what is not. Essentially, don't release a new major until the next branch has been updated and at least compiles.

bnoordhuis commented 9 years ago

I want 4.3 badly to fix many vm bugs #1773. 3.0 now. jsdom wants badly.

I think most of the fixes can be cherry-picked if necessary.

imyller commented 9 years ago

The progress is good. V8 v4.3, v4.4 are a good thing. Push forward :+1:

The bad thing is the resulting regular-user experience if 64K packages stop installing "overnight" after the 3.0.0 becomes the primary version downloadable from iojs.org. Many just get the version offered on the front page and run with that. In the hearts and minds of the regular users this might reflect badly on io.js experience. Majority of users don't care (or know) about V8 or its versioning, I think, so the blame falls on io.js "product".

As a semi-regular user, with professional interest in Node@Embedded platforms, I hope that there would be at least the tools for smooth migration available well before the compatibility breaking core version drops to the masses. This means NAN, FFI and related documentation.

domenic commented 9 years ago

Unless you are going to cherry pick the entire MaybeLocal and Maybe infrastructure, seems unlikely to work.

Besides, we really need to stay on a stable, supported version of V8. Anything else sets a bad precedent.

kkoopa commented 9 years ago

I wonder if you actually can call something that has drastic changes every other day "stable" in the first place.

domenic commented 9 years ago

The hyperbole isn't really conducive to the type of atmosphere we're trying to work in here.

kkoopa commented 9 years ago

It is not hyperbole. In the past couple of weeks I've had to write over 5000 lines of code to deal with changes in V8.

imyller commented 9 years ago

@domenic trying not to be anti-conducive. "Worried" is the key word here. Both the community and core team deeply cares for the io.js/Node and want it to succeed. Thats all.

I think we can all be happy if there are compelling reasons not to worry about fragmentation in node+npm ecosystem etc. If nothing else, it's good to hear core teams thoughts on that.

rvagg commented 9 years ago

@kkoopa

It is not hyperbole. In the past couple of weeks I've had to write over 5000 lines of code to deal with changes in V8.

I feel this pain and am so thankful to have you and the others who have continued the NAN efforts.

I've said it elsewhere but I agree with the sentiment that this isn't sustainable. My hope is that the new working relationships between Node and V8 teams will begin to bear fruit in a less erratic API soon, but it's something we need to monitor and if the V8 team can't stabilise their own internal processes that lead to this kind of drama for us then potential options I see are:

Or combinations of the above. We simply can't keep on with business as usual.

Fishrock123 commented 9 years ago

Yes, breaking every 6 weeks is not exactly "API Stable". What I seem to be hearing so far is that people tend towards wanting node to have API Stability over bug fixes. (Not conclusive, but it seems a general sentiment to me)

kkoopa commented 9 years ago

NODE_SET_METHOD and NODE_SET_PROTOTYPE_METHOD need updating for MaybeLocal.

mikeal commented 9 years ago

I've been listening to the last TSC meeting and heard @rvagg say something that I think I need to address.

Is this just a matter of messaging? Can we just message to people that they will be hitting another break and that they may want to skip this release?

Ok, first of all, this isn't how "messaging" works. If we're on a 6 week release cycle we can't really message anything on a per-release basis that we can reasonably expect most of the people using the release to actually read and understand.

We've been working towards consistent messaging about what you should expect when taking the current release or taking prior releases with LTS support. It's going well but I'd say that less than 5% of our users actually understand that messaging to date and we consistently get the same questions from the community during every major release.

Ok, so, let's say we could effectively message what this release means for the community: this message is pretty garbage. Here's what the message looks like:

We're going to break all your native addons now, and also in 6 weeks. Will they break again in another 6 weeks? Maybe, we don't actually know, and we don't even have a good plan yet about how we're going to make this better.

Here's a question I really need an answer for: What is so valuable about releasing 4.3 right now that it's worth causing so much breakage before we even have a solid plan for alleviating this pain in the future?

What exactly is the rush about? If we want to get people working with 4.3 today so we can get a head start on resolving these breaks let's ship more RC releases. Hell, we could do an even better job of this if we spun up nightly Canary builds of the next branch than we would releasing a 3.0.0 today with 4.3.

It's incredibly damaging to the credibility of our current release lines and the messaging we've been trying to build if we do a release that we effectively tell people not to use. This is what Canary builds are for, we should do those.

Additionally, we should prioritize the planning around the native addons layer and come up with, at minimum, a good story there before we ship breakage this deep in to the community. We have FFI almost ready, we just saw a total re-write of nan, and we've got the Chakra work to consider. This is an ideal time to figure out a solid plan and we're going to need that kind of messaging in order to convince people that this upgrade/re-write of their native addon is likely to be one of the last.

kkoopa commented 9 years ago

I fully agree with @mikeal. I wish I had written that post myself.

piscisaureus commented 9 years ago

@mikeal :clap:

mikeal commented 9 years ago

Besides, we really need to stay on a stable, supported version of V8. Anything else sets a bad precedent.

Why doesn't keeping the next branch along with Canary releases on 4.3 in the interim satisfy this?

domenic commented 9 years ago

@mikeal: So io.js becomes just another Node.js, sticking on outdated and unsupported V8 releases? This is extremely disappointing, speaking for both myself and the Chrome team. We were hoping that io.js would be a server side runtime that, like Chrome, releases every 6 weeks in order to stay secure and up to date.

We definitely shouldn't do a release that we tell people not to use. We definitely do need to work on a better solution longer term. But in the meantime, we can't afford to stagnate. We should release a 3.0.0 now, with V8 4.3; we should then release a 4.0.0 with V8 4.4 in another 4 weeks (or more realistically, 6 weeks, since we seem to lag about two weeks behind V8 due to various issues, including this kind of hand-wringing). We can work toward a layer that better protects us from V8 changes over time, but we cannot react to progress with stagnation.

To answer

Why doesn't keeping the next branch along with Canary releases on 4.3 in the interim satisfy this?

specifically, because the version of io.js released for use will not be using a supported, stable version of V8. Canary releases should be on the next V8, or even the next-next. We haven't gotten the pipeline up and working very well yet, unfortunately, but that doesn't mean we should regress on our plans.

Regarding the OP: @rvagg there is a new 4.3 release that backports the patch you're floating.

mikeal commented 9 years ago

So io.js becomes just another Node.js, sticking on outdated and unsupported V8 releases?

What was that earlier comment you made about hyperbole? :)

We definitely shouldn't do a release that we tell people not to use. We definitely do need to work on a better solution longer term. But in the meantime, we can't afford to stagnate.

First of all, we aren't stagnating, we're still doing regular releases that contain everything except v8 4.3.

Here's the thing, we can't sustain community support for this regular release cycle if most of the ecosystem is broken. If we do two large breaks that require native addon re-writes (not just the regular ol' "bump your version of nan" PR) in as many months I'm afraid that native module authors will simply tell their users to stop upgrading.

Even worse, breaks this frequent make any release unusable for applications with deep dep trees because we're introducing the very real possibility that one dependency will have a version that works with one version and another dependency that works with an entirely different dep ranges, and the user has no single node.js version they can run that supports both dependencies because they've decided to skip over supporting differing releases.

We were hoping that io.js would be a server side runtime that, like Chrome, releases every 6 weeks in order to stay secure and up to date.

Technically we release every week, sometimes more. What we've been trying to do is push a major release with a small set of breaking changes every 6 weeks, similar to Chrome.

Chrome doesn't stop working for 1/3 of the websites on the internet when it does each release but that's essentially what you're asking us to do by accepting this schedule as a matter of policy without consideration for its impact. Chrome would never tolerate this level of ecosystem breakage under any release schedule.

specifically, because the version of io.js released for use will not be using a supported, stable version of V8. Canary releases should be on the next V8, or even the next-next.

This is a good argument for moving the next branch to 4.4 but not for taking 4.3 in the interim. We have a set number of large scale addon breaks the ecosystem will tolerate and we shouldn't waste one of them just because we won't be current with v8 for 6 whole weeks.

bricss commented 9 years ago

Hi, we need to bump V8 version number little bit higher, according to this list -> https://omahaproxy.appspot.com/ latest version is 4.3.61.23.

domenic commented 9 years ago

Thanks for that response. I appreciate your points, and agree with most of them, especially about ecosystem breakage. They do give me pause. And yes, I apologize for the hyperbole; it was out of line.

However, I still think it's too early in the project's lifecycle to set a precedent of this kind of surrender.

What's happened here, in my opinion, is that we're backlogged. We're constantly a few weeks behind Chrome in releases, instead of having a release ready-to-go the moment Chrome stable rolls out. We don't have usable canary releases that people are testing 12 weeks ahead of time, or dev releases people are testing 6-12 weeks ahead of time, or betas people are testing 0-6 weeks ahead of time, like Chrome does. If we had that kind of pipeline, then hopefully the process of rolling to a new release would cause much less breakage because people would have had 12 weeks or so to accept pull requests and coordinate. And we haven't even started on a project of getting people to loosen their NAN restrictions to ^ instead of ~, or working on a better protection layer as @rvagg alludes to.

But that's on us. I think we should take it in stride, and do the releases anyway, and take it as incentive to do better in the future. For example, after releasing 3.0.0 with 4.3, we should definitely get started on a parallel next branch with 4.4, and we should aim to be in a state where we feel comfortable releasing it as 4.0.0 tomorrow if need be, so that when Chrome 44 rolls out we don't have to track down V8 bugs causing test failures or have a debate about whether it's really OK to do a release.

Here's the thing, we can't sustain community support for this regular release cycle if most of the ecosystem is broken. If we do two large breaks that require native addon re-writes (not just the regular ol' "bump your version of nan" PR) in as many months

Can someone quantify these? My understanding of 3.0.0 is that all the breakage is contained there, and 4.4 doesn't have much beyond bump-your-nan-version. There will be more breakage down the like (maybe 4.5? 4.6?) when the non-Maybe versions of the methods are removed, but we can get out ahead of that if we're smart.

sam-github commented 9 years ago

and 4.4 doesn't have much beyond bump-your-nan-version

Just a note - from a user's perspective, I don't care if my deep sub-dependencies broke because someone wasn't use ^ and has to make a small change to a package file, or if someone has to take an hour and modify their code because NAN was forced to make a breaking API change. Either way - my app doesn't install and there ain't much I can do about it other than roll back to the last io.js release, its exactly equivalent.

Which is partly why its so puzzling to me that nan isn't comitting to semver and recommending use of ^. Its one thing to have breakage forced upon you by V8, its another to tolerate and expect breakage to occur even though NAN's existing API was theoretically capable of hiding the V8 change.

Qard commented 9 years ago

@sam-github :+1:

The vast majority of code node devs use is written by someone else, and they likely have little control over if/when that dependency updates. In an ideal world, you'd just take the time to write a PR and it'd get merged and released right away. But, for much of the community, that is a fantasy. People have their own priorities; sometimes writing or reviewing that PR takes a back seat to the work they need to do to pay the bills. Unfortunately that also reflects badly on the people the community perceives as responsible for allowing that breaking change to happen.

Most devs don't care about the shiny and new things, they just care that their code works and won't need to be rewritten next week.

kkoopa commented 9 years ago

4.4 is breaking external data properties, 4.5 is breaking weak callbacks, 4.6 will definitely break something else.

It is the responsibility of io.js to have a clear path forward to V8 master on each release. Code that runs on io.js version X should be guaranteed to run on version X + 2.

On May 28, 2015 7:44:16 PM EEST, Domenic Denicola notifications@github.com wrote:

Thanks for that response. I appreciate your points, and agree with most of them, especially about ecosystem breakage. They do give me pause. And yes, I apologize for the hyperbole; it was out of line.

However, I still think it's too early in the project's lifecycle to set a precedent of this kind of surrender.

What's happened here, in my opinion, is that we're backlogged. We're constantly a few weeks behind Chrome in releases, instead of having a release ready-to-go the moment Chrome stable rolls out. We don't have usable canary releases that people are testing 12 weeks ahead of time, or dev releases people are testing 6-12 weeks ahead of time, or betas people are testing 0-6 weeks ahead of time, like Chrome does. If we had that kind of pipeline, then hopefully the process of rolling to a new release would cause much less breakage because people would have had 12 weeks or so to accept pull requests and coordinate. And we haven't even started on a project of getting people to loosen their NAN restrictions to ^ instead of ~, or working on a better protection layer as @rvagg alludes to.

But that's on us. I think we should take it in stride, and do the releases anyway, and take it as incentive to do better in the future. For example, after releasing 3.0.0 with 4.3, we should definitely get started on a parallel next branch with 4.4, and we should aim to be in a state where we feel comfortable releasing it as 4.0.0 tomorrow if need be, so that when Chrome 44 rolls out we don't have to track down V8 bugs causing test failures or have a debate about whether it's really OK to do a release.

Here's the thing, we can't sustain community support for this regular release cycle if most of the ecosystem is broken. If we do two large breaks that require native addon re-writes (not just the regular ol' "bump your version of nan" PR) in as many months

Can someone quantify these? My understanding of 3.0.0 is that all the breakage is contained there, and 4.4 doesn't have much beyond bump-your-nan-version. There will be more breakage down the like (maybe 4.5? 4.6?) when the non-Maybe versions of the methods are removed, but we can get out ahead of that if we're smart.

Reply to this email directly or view it on GitHub: https://github.com/nodejs/io.js/issues/1807#issuecomment-106477323

bnoordhuis commented 9 years ago

What is so valuable about releasing 4.3 right now that it's worth causing so much breakage before we even have a solid plan for alleviating this pain in the future?

What's the alternative? Stop upgrading? Damned if you, damned if you don't.