Latest NodeJS 4.9.0, 6.14.0, 8.11.0, 9.9.0 and 9.10.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3

[Flarna]

• Version: 4.9.0, 6.14.0, 8.11.0, 9.9.0 and 9.10.0
• Platform: PPCLE
• Subsystem: build

I found that NodeJS 9.9.0 is not running on our PPCLE machine, NodeJS 9.8.0 was fine.

Problem reported: /lib64/libstdc++.so.6: version 'GLIBCXX_3.4.20' not found

I did some objdump and I found following symbol in 9.9.0 which was not used in 9.8.0:

0000000000000000      DF *UND*  0000000000000000  GLIBCXX_3.4.20 _ZSt24__throw_out_of_range_fmtPKcz
demangled:
0000000000000000      DF *UND*  0000000000000000  GLIBCXX_3.4.20 std::__throw_out_of_range_fmt(char const*, ...)

Is a new compiler version used starting with 9.9.0? And is this intended?

Some details about our machine: Red Hat Enterprise Linux Server release 7.3 ldd (GNU libc) 2.17 gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)

[gireeshpunathil]

/cc @nodejs/build

[gibfahn]

cc/ @jBarz @mhdawson @refack , but I think we have moved up to gcc 4.9 from gcc 4.8 recently.

See also https://github.com/nodejs/build/issues/1020

[Flarna]

I think use of gcc 4.9 could explain this. But I don't see this dependency for x86 binaries. Are x86 binaries still built with 4.8?

[bnoordhuis]

Hm, the problem is this (from here):

GCC 4.8.0: GLIBCXX_3.4.18, CXXABI_1.3.7
GCC 4.8.3: GLIBCXX_3.4.19, CXXABI_1.3.7
GCC 4.9.0: GLIBCXX_3.4.20, CXXABI_1.3.8  <- bump

I suppose the solution is to revert the 4.8->4.9 upgrade in <= v9.x.

[jBarz]

Using rpath option at the link step will force the binary to use the 4.9 libraries? We should be using the following in the environment.

export CC=/usr/bin/gcc-4.9 export CXX=/usr/bin/g++-4.9 export LINK=/usr/bin/g++-4.9 export LDFLAGS=“-Wl,-rpath,$(dirname $($CC --print-file-name libgcc_s.so))”

[Flarna]

@jBarz The problem is that there are no 4.9 libs for PPCLE Red Hat Enterprise Linux Server 7.x.

Seems 9.9.0 was at least partly built with GCC 4.9.4:

$ readelf -p .comment portable-nodejs-linux-ppcle_64-9.9.0/node 

String dump of section '.comment':
  [     0]  GCC: (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4
  [    2b]  GCC: (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

$ readelf -p .comment portable-nodejs-linux-ppcle_64-9.8.0/node 

String dump of section '.comment':
  [     0]  GCC: (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

As reference for x86 I get:

$ readelf -p .comment portable-nodejs-linux-x86_64-9.9.0/node 

String dump of section '.comment':
  [     0]  GCC: (GNU) 4.4.7 20120313 (Red Hat 4.4.7-18)
  [    2d]  GCC: (GNU) 4.8.2 20140120 (Red Hat 4.8.2-15)

$ readelf -p .comment portable-nodejs-linux-x86_64-9.8.0/node 

String dump of section '.comment':
  [     0]  GCC: (GNU) 4.4.7 20120313 (Red Hat 4.4.7-18)
  [    2d]  GCC: (GNU) 4.8.2 20140120 (Red Hat 4.8.2-15)

[Flarna]

I suppose the solution is to revert the 4.8->4.9 upgrade in <= v9.x.

Does this indicate that NodeJS 10 will use GCC 4.9 and as a result doesn't work out of the box on our Red Hat Enterprise Linux Server 7.x?

[bnoordhuis]

That seems likely.

[gibfahn]

Does this indicate that NodeJS 10 will use GCC 4.9 and as a result doesn't work out of the box on our Red Hat Enterprise Linux Server 7.x?

V8 has started to use C++14 features (AIUI) which will necessitate gcc 4.9. However if it's not straightforward to get the gcc 4.9 libstdc++ on RHEL 7, then we probably need to document how it should be done for Node users.

I guess alternatively it might be a good time to consider statically linking against libstdc++.

[bnoordhuis]

I guess alternatively it might be a good time to consider statically linking against libstdc++.

I've thought about that (but libc++ because license) but I foresee an endless stream of issues with native modules.

For example, consider a module foo that links against /usr/lib/libfoo.so. If the library links against a different C++ runtime, the spectrum of things that can go wrong runs from link-time errors to segfaults and data corruption.

Longer term it might be worth it but it's probably not going to be an easy transition. I don't know if we could get it ready in time for Node.js 10 and have all the kinks worked out.

[cosinusoidally]

I guess alternatively it might be a good time to consider statically linking against libstdc++.

I've thought about that (but libc++ because license) but I foresee an endless stream of issues with native modules.

Don't you build your x86/x64 release binaries with some version of devtoolset on CentOS? If so, you are already statically linking to parts of libstdc++ on x86/x64. In devtoolset its copy of libstdc++.so looks something like:

cat /opt/rh/devtoolset-4/root/usr/lib/gcc/x86_64-redhat-linux/5.3.1/libstdc++.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
INPUT ( /usr/lib64/libstdc++.so.6 -lstdc++_nonshared )

which is a linker script that dynamically links to the system libstdc++.so (which for CentOS 6 will be the one from GCC 4.4) and then statically links in any missing libstdc++ symbols using libstdc++_nonshared.a (the symbols in libstdc++_nonshared.a are the ones introduced by the newer version of GCC that ships with a given devtoolset release). This allows binaries to run on CentOS 6 without needing to ship a newer version of libstdc++.so with your program.

Maybe the solution for PPCLE RHEL would be to build with a devtoolset (if available)?

[Flarna]

Toolchain seems to be unchanged; 9.10.0 seems to be like 9.9.0. I already ordered an Ubuntu 16.04 machine to run test with NodeJs >=9 tests :grinning: For building our native addon we keep the old one to avoid raising glibc requirements. We don't use node in build step so it's no issue there.

So from my point of view this issue can be closed as the root cause is understood and it's clear that there is no way around for NodeJS 10 to raise the requirements. I don't care much about the final NodeJS 9 releases.

[Flarna]

Just found that all fresh NodeJS versions are effected including 4.9.0, 6.14.0, 8.11.0.

[mhdawson]

Ok, the problem seems to be that we were staging the 4.9 compiler and I thought the default would stay as 4.8, but it looks like it is actually 4.9.,

I'll add the selection logic and we can validate that it resolves the issue on 4.X and 6.x. The lowest stated version was 4.9 for 8.X so the selection logic uses 4.9 for 8.X and above. However, we'll have to consider that again given the info in this thread.

Did anybody check if this affects x86 as well ?

[Flarna]

x86 is not effected

[mhdawson]

@Flarna so on x86 4.9 is compatible with 4.8 but on PPCLE that is not the case?

[mhdawson]

@Flarna is there a version newer than 4.9 that is easier to get on RHEL 7.X ?

[Flarna]

so on x86 4.9 is compatible with 4.8 but on PPCLE that is not the case?

To me it looks more like x86 is still built with GCC 4.8. See my https://github.com/nodejs/node/issues/19530#issuecomment-375595253 above where I used readelf to get GCC infos.

is there a version newer than 4.9 that is easier to get on RHEL 7.X ?

Honestly speaking I don't know what is available as I'm just an user.

In our addon build we use GCC 6.4.0 but it's a self built GCC patched for our needs. We link statically to avoid issues like this but I don't think our solution is directly applicable to NodeJS. In our addon we take care to not expose any C++ symbol but NodeJS can't do that as it has to export V8 API.

[mhdawson]

@Flarna can you test out this version:

https://nodejs.org/download/test/v6.14.0-test2c93af2da3/node-v6.14.0-test2c93af2da3-linux-ppc64le.tar.xz

It is the 6.14.0 release rebuilt with the detection logic so that it built with the 4.8 compiler.

If that works then we can decide if we should change the logic for 8.x and work on respining the builds.

[MylesBorins]

Should we be planning for a member patch release for all lines?

[Flarna]

@mhdawson Yes, this version works.

[mhdawson]

@Flarna thanks

@MylesBorins, not sure what you mean by a member patch release, but what we need is to rebuild the le releases for 4, 6, and 8 (for now I'm just going to make 8 build with 4.8 instead of 4.9 until we figure out the story across platforms).

There are no code changes so I think we could just rebuild and republish for ppcle unless that's not something we want to do.

I still need to do one update to get it right for 8 and then we could rebuild with the ppcle job I created.

[mhdawson]

@MylesBorins can you approve https://github.com/nodejs/build/pull/1201. Once that lands and I validate with a build in test, we can respin the builds.

[richardlau]

We generally don't replace already published builds to avoid confusion. Usual practice is to do a new patch release.

[MylesBorins]

+1 to what Richard said. We've already published and signed the Shas for the current releases, this would require a new patch release.

On Thu, Mar 29, 2018, 1:42 PM Richard Lau notifications@github.com wrote:

We generally don't replace already published builds to avoid confusion. Usual practice is to do a new patch release.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nodejs/node/issues/19530#issuecomment-377315656, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecV99k6Tbp9NHzYgigTpINmR49R-GTks5tjR0jgaJpZM4S2zJ- .

[mhdawson]

ok so it will then require releases across 4, 6, and 8

@MylesBorins can you approve nodejs/build#1201. I'll then land and test and then we'll be ready to do the new releases.

[mhdawson]

That PR does not actually change any code in node, just a script which controls which compiler is used on LE.

[mhdawson]

or @richardlau can you approve nodejs/build#1201

[mhdawson]

Ideally the patch releases would only have the change of compiler on PPC LE since the previous release was a security release.

[richardlau]

Sorry, been vomiting for the last hour 🤢 . I can see how prior to https://github.com/nodejs/build/pull/1201 Node.js 8.x was affected, but how were 4.x and 6.x affected?

[MylesBorins]

I've opened release PRs for all lines. Will close this after they land.

Will someone be able to test the builds before we push the release?

[mhdawson]

I've found a RHEL 7.1 machine and will test them there.

[mhdawson]

@richardlau answered your question in nodejs/build#1201

[mhdawson]

Tested the new versions, they all run -version as well as a starting the console and being able to print 'test'. I think they are good.

[MylesBorins]

New versions are released. Closing the issue. Please let me know if it isn't fixed and I'll reopen

[Flarna]

I can confirm that all four NodeJs versions run fine on our RedHat Enterprise 7.3.

[mukeshkmr776]

@mhdawson I am having same problem for NodeJS-10.15.3 for PPCLE (RedHat 7.x). Really exhausted trying finding solution for this.

Please confirm if same problem exists for NodeJS-10.15.3 also.

And, suggest some way to make NodeJS-10.15.3 work for RHEL 7.x (PPCLE) (Power8)

It is making me cry :'(

[mukeshkmr776]

@mhdawson @MylesBorins

[Flarna]

@mukeshkmr776 NodeJs 10 and newer uses gcc 4.9 and as a result it wont run on these old platforms. see https://github.com/nodejs/node/issues/19530#issuecomment-375595761

[mukeshkmr776]

@Flarna Thanks for confirming this. Now, I am not getting gcc version >=4.9.x even after updating my server rpm through yum. Could you please provide me the binary libstdc++.so.6 or complete rpm or a link to it so that I can move ahead. :(

One observation, then how come Nodejs 10.15.3 is working on my another system of RHEL 7.x (ppc) with gcc-4.8.x ???? :/

[Flarna]

For Node.Js < 10 this was fixed in node builds as they stepped back to gcc 4.8. We haven't found a solution for Red Hat, we moved to Ubuntu instead.

One observation, then how come Nodejs 10.15.3 is working on my another system of RHEL 7.x (ppc) with gcc-4.8.x ???? :/

Sorry, don't know but maybe ppc doesn't require a newer libstdc++ but pccle does. As far as I remember we had no problems on x86 either that time, only ppcle (we never used ppc).

[mukeshkmr776]

@Flarna So what should I do then? :| Nodejs v10 working on ppc64, but on ppc64le. Though both had same gcc 4.8.x versions :/

[Flarna]

NodeJs v10 uses gcc 4.9.4 not 4.8. I would say if it works on ppc64 it's just luck. As mentioned above x86 worked also for me that time. Moving from gcc 4.8.3 to 4.9 moved GLIBCXX from 3.4.19 to 3.4.20. As far as I understood the versioning of glibcxx the exported symbols are versioned. Only changed exports get a new version and which ones are effected depends on the platform and which ones are actually used depends on the application.

I think you have just two options (I assume RedHat does not provide anything helpful here):

• try to build gcc 4.9 toolchain by yourself, I expect it's not easy...
• move to a different system (like we did)

Even if you get Node 10 running it will be not a future prove solution as NodeJs 12 has moved to gcc 6.3 (GLIBCXX 3.4.22) as far as I know.

[justinreock-roguewave]

Was this fix applied to the build process for the ppc64le binaries? https://github.com/nodejs/build/issues/1548

I've been looking into this issue the last couple of days and it really is a mystery that the build for ppc64le is linked to 3.4.20. x86_64 builds are linked to the same version of gcc and work fine with 4.8.5, which provides 3.4.19:

`

`

It seems to me like a bug in the build process that the ppc64le binaries are linking against 3.4.20.

This only affects 10.x and later versions, as we know. I'm running a build of 10.15.2 manually on a ppc64le machine and seeing if I can force it to use gcc 4.8, will see if that does anything, but in the mean time can we check to see if the fix applied above was applied to the ppc64le build pipeline?

[justinreock-roguewave]

Further evidence that this is a broken build process. When I dump out the comment section of the binaries for ppc vs x86 I get very, very different results:

x86_64: [root@personal-justin-amqload1 bin]# readelf -p .comment node

String dump of section '.comment': [ 0] GCC: (GNU) 4.4.7 20120313 (Red Hat 4.4.7-23) [ 2d] GCC: (GNU) 6.3.1 20170216 (Red Hat 6.3.1-3)

[root@personal-justin-amqload1 bin]#

ppc64le: [jreock@dc3-zsnode1 bin]$ readelf -p .comment node

String dump of section '.comment': [ 0] GCC: (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4 [ 2b] GCC: (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

[jreock@dc3-zsnode1 bin]$

[sam-github]

We are in process of beginning to build 10x/ppcle on centos7 with devtools-6, which should extend the backwards compatibility of the official binaries to even older machines than those currently supported. See https://github.com/nodejs/build/pull/1907 among others. We hope this will be released in the next month or so.

[justinreock-roguewave]

Thanks for the reply, Sam – this is a great approach. Will look forward to the push, let me know if I can help at all!

-Justin

From: Sam Roberts notifications@github.com Reply-To: nodejs/node reply@reply.github.com Date: Thursday, September 5, 2019 at 12:22 PM To: nodejs/node node@noreply.github.com Cc: Justin Reock Justin.Reock@roguewave.com, Comment comment@noreply.github.com Subject: Re: [nodejs/node] Latest NodeJS 4.9.0, 6.14.0, 8.11.0, 9.9.0 and 9.10.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3 (#19530)

We are in process of beginning to build 10x/ppcle on centos7 with devtools-6, which should extend the backwards compatibility of the official binaries to even older machines than those currently supported. See nodejs/build#1907https://github.com/nodejs/build/pull/1907 among others. We hope this will be released in the next month or so.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/nodejs/node/issues/19530?email_source=notifications&email_token=AFUQ75SRYPDIBKY7DUWJPQTQIEW33A5CNFSM4EW3GJ7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD57XFKA#issuecomment-528446120, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFUQ75WKZOCLPFTIBDBZSGTQIEW33ANCNFSM4EW3GJ7A.

[sam-github]

Once we have builds, it would be useful to confirm for us they help your situation.

[justinreock-roguewave]

Absolutely! I'll be keeping an eye on this and will confirm functionality.

[mukeshkmr776]

Please update once this is fixed for Nodejs-v10.x on PPC64LE (RHEL-7.x). Eagerly waiting on this