Dropping 32-bit builds

[gibfahn]

This came out of discussion of https://github.com/nodejs/build/pull/809 in the last WG meeting (https://github.com/nodejs/build/issues/875, minutes at https://github.com/nodejs/build/pull/881). Discussion paraphrased below.

Basically updating CentOS 6 to gcc 4.9.3 requires devtoolset-6, which means that we can no longer do 32-bit xLinux builds.

It's difficult to know how many people are currently using x86_64 32-bit Node. The metrics show it looking pretty stagnant, but it's difficult to know how many of the downloads are from CI.

@rvagg suggested that the best way to find out would be to drop 32-bit builds in an odd-numbered release and see how many people complain, that way we have the option of adding them back in a minor version. The question was whether 9.x is too soon to do this.

Rod also said that he thought CentOS6 and Ubuntu 14 no longer had a 32-bit version, meaning that anyone relying on a 32-bit OS had already been left behind by Node 8.x. If we could confirm that this is true for most Linux and Windows versions then that would make the decision much easier.

Todo (if we decide to drop 32-bit)

• Update the jobs to skip 32-bit Intel for Node >8.x
• Update the website to handle the lack of 32-bit builds
• Update the Supported Platforms list to remove x86.

[gibfahn]

More info on the state of play at the OS level would definitely be useful (i.e. when/if major distros stopped supporting 32-bit OSs, i.e. who we'd be breaking).

• Linux: maybe @rvagg or @chrislea would know off the top of their heads
• Windows: @joaocgreis and @refack
• SmartOS: @nodejs/platform-smartos would be good to know your thoughts, I assume we don't have to drop 32-bit on SmartOS, but if we're doing it elsewhere it might be a good time to do it here too.

[refack]

Windows 10 still supports 32bit and "cross compiling" from a 64bit host to a 32bit target is also fully supported. Windows Server 2008R2 was not released as 32bit host, but Windows has side-by-side technology (WoW64) that allows 32bit applications to run natively on 64bit hosts, so there should not be an external limitation for building, testing and running node32 for the foreseeable future.

BTW: AWS/Azure/GCP doesn't rent 32bit VMs of our lowest supported Windows (Server 2008R2).

[chrislea]

There is absolutely still support for 32bit CentOS 6 / RHEL 6 Linux out there. Also, all Debian and Ubuntu releases still have 32bit versions, and Fedora still makes 32bit releases as well.

Red Hat dropped 32bit support with RHEL 7.

I don't have much of a sense of how many people are using 32bit builds in production. If I had to guess (emphasis mine there), I doubt many people are running 32bit Node builds on production servers, and I suspect the biggest impact might be not so much on production servers, but instead on appliances. For example, here's output from a Synology NAS that I have at home:

chl@DSPlay01:~$ uname -a
Linux DSPlay01 3.2.40 #15152 SMP PREEMPT Fri Sep 1 11:13:20 CST 2017 i686 GNU/Linux synology_evansport_214play

And yes, Node is an installable package should I choose to put it on this device. A lot of these appliance gizmos use 32bit chips because they don't have much memory and want to keep costs low, so older 32bit Atom or Celeron processors make sense.

Now, having said all that, if we assume I'm right, then I don't think we need to be too concerned about anything other than making sure that Node will in fact build on a 32bit system with a modern enough compiler, because from what I can tell almost all of these appliances have their own packaging formats and tend to build the things they need for those packages themselves.

[seishun]

@gibfahn I don't see any reason to drop 32-bit builds on Windows.

@chrislea

then I don't think we need to be too concerned about anything other than making sure that Node will in fact build on a 32bit system with a modern enough compiler

That shouldn't be a problem. We could just continue running CI jobs on 32-bit platforms where it's easy enough to install a supported compiler, e.g. Ubuntu 14.04.

[joaocgreis]

We shouldn't drop Windows 32-bit support. While our deps have support for it, it'll be easy for us to maintain. Windows and Visual Studio have full support for 32-bit.

[bnoordhuis]

Apropos Fedora, i686 is in a state of disrepair and has been for years: https://lwn.net/Articles/728207/

There is a Special Interest Group since a few months that fixed some things but I think it's safe to say no one runs Fedora i686 in production and it's not worth spending time on.

[chrislea]

Unfortunately I have seen people running 32bit Fedora out there in the wild (same for CentOS 6 and occasionally an older Debian / Ubuntu release). But generally I feel like all the major distros want to drop it and are sort of looking at each other waiting to see who does it first. I know Arch Linux (not one of the major ones, but important among the more hacker / bleeding edge types) is dropping it officially in November of this year.

So I still feel pretty comfortable with what I said above, which is that I think we'd be okay if we continue to "support" it in the sense that we make sure it will in fact build on a 32bit x86 machine, but we stop making official release tarballs for it.

[gibfahn]

So maybe the answer is to drop it to Tier 2 or Experimental rather than dropping completely.

[jasnell]

Some informal research shows that there are still a fair number of users using 32-bit builds on smaller IoT devices. I doubt this is a large group, but it's still worth bearing in mind.

I'm going to tag this with a tsc-review label.

[chrislea]

Yes @jasnell if you'd like my $0.02 on the "devices" thing (I'm sure you're just dying to hear) it's here.

[gibfahn]

@chrislea wrong link? That seems to point to my comment above.

[chrislea]

My bad, try this.

[seishun]

It's been a week and I see no objections to dropping public 32-bit Linux builds. Perhaps we can proceed with that for now and leave the discussion about dropping 32-bit support (or downgrading it to Experimental) for later?

[refack]

@nodejs/release did discuss this at the last meeting. @gibfahn what was the decision (I remember something about IoT, and maybe stop releasing, but keep testing..)?

[gibfahn]

@nodejs/release did discuss this at the last meeting.

Discussed here, there wasn't a decision, but @jasnell mentioned that he believed he knew of some embedded devices that were Intel 32-bit Linux, and that he'd had requests from people for us to not drop 32-bit builds. So more info on that would be good. James did say that he thought continuing to test but not doing releases sounded reasonable.

It's been a week and I see no objections to dropping public 32-bit Linux builds.

This was raised for tsc-review, but it looks like it got missed at the last meeting, https://github.com/nodejs/TSC/issues/359 (or at least it's not in the minutes).

[gibfahn]

Perhaps we can proceed with that for now and leave the discussion about dropping 32-bit support (or downgrading it to Experimental) for later?

That seems like a good idea.

So, does anyone have an objection to us ceasing to do 32-bit xLinux releases from Node 9.x onwards? We will continue to run CI. cc/ @nodejs/build @nodejs/lts @nodejs/tsc

[mscdex]

Might want to update the original post if this is about Intel 32-bit only and not other platforms (e.g. ARM).

[gibfahn]

Might want to update the original post if this is about Intel 32-bit only and not other platforms (e.g. ARM).

It was originally supposed to be a more general discussion (including ARM). The xLinux discussion is just the most pressing matter.

[mscdex]

What is "xLinux?" Is it an IBM thing?

[gibfahn]

What is "xLinux?" Is it an IBM thing?

x86_64 Linux, as opposed to aLinux (ARM Linux), pLinux (Power Linux), and zLinux (z Linux).

Probably used more often in IBM, as we deal with more architectures than most, but I don't think it's exclusive.

[chrislea]

Ubuntu is going to stop making 32bit desktop ISOs with their next release, just FYI.

[rvagg]

That full context on the Ubuntu decision is actually pretty informative about their perspective on the future of i386

[rvagg]

https://lists.ubuntu.com/archives/ubuntu-release/2017-September/004212.html

[seishun]

Still no objections. What next?

[gibfahn]

No objections raised by anyone in Build or TSC AFAICT, so I think this is agreed unless someone wants to put it on tsc-agenda and discuss there. @jasnell is that something you want to do?

Otherwise this is done, it's just a question of someone adding the correct if statements to the release build job.

@rvagg @joaocgreis you've done this more than most I think, if I come up with a diff can you review?

[seishun]

Do https://github.com/nodejs/build/pull/797 and https://github.com/nodejs/build/pull/809 need to wait until the release build job change is implemented?

[refack]

AFAICT #797 can land (we can build & test) and it's not used for releases. https://github.com/nodejs/build/pull/809 is problematic since ATM we can't even build for testing on x32.

[seishun]

809 is problematic since ATM we can't even build for testing on x32.

There are other 32-bit testing machines besides CentOS 6.

[refack]

809 is problematic since ATM we can't even build for testing on x32.

There are other 32-bit testing machines besides CentOS 6.

Yeah but we still want to CI on CentOS6.i686 to make sure people are able to at least build their own. From experience if we don't CI test a configuration it will rot. BTW: I'm now trying to build gcc for i686 and also trying to use a native i686 VM

[seishun]

I doubt anyone is going to build gcc to build the latest Node.js on 32-bit CentOS 6, so going to such lengths on our side seems unnecessary. Not my call though.

[gibfahn]

Yeah but we still want to CI on CentOS6.i686 to make sure people are able to at least build their own. From experience if we don't CI test a configuration it will rot.

This is definitely something we should discuss, the current proposal is not to build releases, but to continue to run CI, but we haven't discussed what we're going to say we support (and what we're going to test).

• CentOS 6 is obviously an issue due to the devtoolset problem.
• RHEL 7 doesn't support 32-bit (https://github.com/nodejs/build/issues/885#issuecomment-330027207)
• Ubuntu 14 and 16 should work fine (https://github.com/nodejs/build/issues/885#issuecomment-330027207)

As @chrislea points out, the problem is more likely to be appliances (https://github.com/nodejs/build/issues/885#issuecomment-330027207), in which case the key is likely to just be trying to maintain relationships with the package builders, and specific OS support probably won't be so important. That sounds like the type of system @jasnell is talking about as well.

If we can get CentOS 6 support via https://github.com/nodejs/build/pull/809#issuecomment-333325561 then why not continue to run CI, but if we can't I don't think it's a huge issue.

[refack]

First to frame the convo: I'm assuming we're talking about node >= 9.0.0.

If we can get CentOS 6 support via #809 (comment) then why not continue to run CI, but if we can't I don't think it's a huge issue.

As I wrote above, if we don't run a platform in CI, IMHO we can't say it's supported (probably not even experimental).

Whether support on Debian based Linux is transitive to RHEL based, I don't know. Maybe we should split the two and state:

System	Support type	Version	Architectures
GNU/Linux (Debian)	Tier 1	kernel >= 2.6.32, glibc >= 2.12	x86, x64, arm, arm64
GNU/Linux (RHEL)	Tier 1	kernel >= 2.6.32, glibc >= 2.12	x64, arm, arm64
GNU/Linux (RHEL)	experimental	kernel >= 2.6.32, glibc >= 2.12	x86

[seishun]

As I wrote above, if we don't run a platform in CI, IMHO we can't say it's supported (probably not even experimental).

It's not explicitly stated anywhere that CentOS 6 is supported, and there are many implicitly supported Linux distros that we don't run in CI, even major ones: Arch, Gentoo, openSUSE...

Whether support on Debian based Linux is transitive to RHEL based, I don't know.

What does "transitive" mean here?

[gibfahn]

Given the definition of Experimental:

Experimental: May not compile reliably or test suite may not pass.

I think that calling x86 Experimental in Node >=9.x is perfectly reasonable. We'll test on the platforms we can, but no guarantees.

I don't think specifying distros is necessary, there are a tonne of distros we don't test on, we just assume that it works, and accept patches from people who find bugs.

To be clear, if there's an easy way to test on CentOS 6 then great, let's do it. But there are lots of other issues with our CI that I would say are higher priority than getting CentOS 6 32-bit builds working.

[refack]

Ok. I'm convinced. So will the table be?

System	Support type	Version	Architectures
GNU/Linux	Tier 1	kernel >= 2.6.32, glibc >= 2.12	x64, arm, arm64
GNU/Linux	experimental	kernel >= 2.6.32, glibc >= 2.12	x86

[seishun]

Does it really have to be experimental? It will still be tested on ubuntu1404-32, ubuntu1604-32 and debian8-x86.

[gibfahn]

Okay, so I think we should just be able to add the skip under this existing one:

if [[ $ARCH =~ s390x && ${NODE_VERSION:0:1} -lt "6" ]]; then
  echo "Not building $disttype on linuxOne slave $NODE_NAME as only supported on v6.x and later"
  exit 0
f

This code should work:

NODE_MAJOR_VERSION="$(echo "$NODE_VERSION" | cut -d . -f 1)"

if [[ "$NODE_MAJOR_VERSION" -gt 8 && "$nodes" =~ centos.*-release-32 ]]; then
  echo "Not building $disttype on 32-bit slave $NODE_NAME for Node 9.x and above"
  exit 0
fi

I'd also recommend changing anything with ${NODE_VERSION:0:1} in it to $NODE_MAJOR_VERSION, as otherwise everything will break with 10.0 (that'll evaluate to 1).

cc/ @joaocgreis @rvagg @mhdawson @jbergstroem PTAL

[joaocgreis]

Regex should probably be centos.*-release-32. Other than that looks good, including the change to $NODE_MAJOR_VERSION. I'd recommend to do this change when there are no releases pending soon, so that it does not interfere (this usually just means don't touch it on Tuesdays). You can test it right away with a test build and then check the nightlies in the day after.

[srl295]

FWIW ICU dropped all 32 bit binary builds after v57.1, March 2016—we continue to build against i386 thanks to the GCC compile farm, because 32 bitness is easy to break with data structs and such.

if we don't CI test a configuration it will rot.

In my opinion, we should keep at least one 32 bit x86 system of any sort, at least as long as v8 is going to support it, to prevent rot.

[refack]

Is there a way for us to host a CentOS6.i686 (a.k.a x32) machine anywhere. Maybe on Azure?

[chrislea]

@refack I think the easiest way to build for 32bit CentOS 6 would be to use mock on a modern version of Fedora. It's a bit of a hack but it should be workable.

You can use mock to create a 32bit CentOS chroot with the devtoolset-6 toolchain installed, then chroot into it and build there.

[joaocgreis]

Already landed the NODE_MAJOR_VERSION line because of https://github.com/nodejs/build/issues/965. The rest of it can probably go together with https://github.com/nodejs/build/issues/957 when there are no releases happening soon.

[seishun]

I see a 32-bit Linux release here, so what's the plan now?

[refack]

I see a 32-bit Linux release here, so what's the plan now?

That's an oversight...

[refack]

Cross-ref: https://github.com/nodejs/nodejs.org/pull/1446 Cross-ref: https://github.com/nodejs/node/pull/16723

[rvagg]

Another one bites the dust https://www.archlinux.org/news/the-end-of-i686-support/

[refack]

Both downstream issues have landed: d/l: 32bit linux is experimental nodejs/nodejs.org#1446 meta: 32 bit linux is experimental nodejs/node#16723

The request was to keep the undocumented/unsupported testing and binaries coming, for as long as we can, but we are no longer "obligated" to do so.

[seishun]

I thought the idea was to drop 32-bit Linux binaries in 9.x to see how many people complain?

[gibfahn]

I'd like us to keep doing nightlies and stop doing release binaries.

[refack]

I thought the idea was to drop 32-bit Linux binaries in 9.x to see how many people complain?

👍

I'd like us to keep doing nightlies and stop doing release binaries.

It's mixed message, but I guess it's the best litmus test. I'll look into how to do that.