Binaries for sunos do not work on Solaris 11.2 - 'GLIBCXX_3.4.20' not found

[GeorgeBailey]

We're increasing our use of Solaris 11.2 X86. The only snag right now is staying compatible with the latest Node.JS releases.

wget http://nodejs.org/dist/v4.2.0/node-v4.2.0-sunos-x64.tar.gz
tar xoZf node-v4.2.0-sunos-x64.tar.gz
node-v4.2.0-sunos-x64/bin/node

I get this for both x86 and x64 binaries.

ld.so.1: node: fatal: libstdc++.so.6: version 'GLIBCXX_3.4.20' not found (required by file node-v4.2.0-sunos-x64/bin/node)
ld.so.1: node: fatal: libstdc++.so.6: open failed: No such file or directory
Killed

Edit: We can now readily build from src on Solaris 11.2 with minor tweaks after downloading v4.2.1.

This issue should now only reference the desire of a pre-compiled that is compatible with Solaris 11.2.

[mscdex]

/cc @nodejs/platform-solaris

[alexijs]

It may not be the best solution, however you can tell the runtime linker to ignore the shared library versions.

SunOS prod 5.11 11.2 i86pc i386 i86pc

IT DOES SIMPLY START IT AND NOT ALLOWING NETWORK/SOCKET CONNECTIONS

[GeorgeBailey]

I can confirm that adding LD_NOVERSION=1 at runtime as @alexijs suggested allows startup, but results in EPROTOTYPE error on TCP or UNIX socket functions. (I did successfully use child processes, crypto.randomBytes.)

Referring to my previous issue, I was getting EPROTOTYPE in v0.12 pre-built binaries as well. This was resolved by cgalibern in a fork. (though I was still getting ECONNREFUSED on outbound TCP)

I wonder whether the same change would resolve EPROTOTYPE in v4.2.0 or if it is a different issue entirely.

[GeorgeBailey]

On mine, I used pkg install gcc which through the dependency tree includes gnu-binutils and I believe it also includes gnu-make. I don't know which compiler is the best to use, but what I really want is to be compatible with both Solaris 10 and Solaris 11. :-)

I don't know what Solaris 10 compatibility would entail, so let's just get Solaris 11 compatibility for now.

[jbergstroem]

@alexijs: CC=/usr/bin/gcc ./configure should work fine if you want to switch compiler

[misterdjules]

There are two different categories of problems mentioned in this issue:

1. Running the binaries downloaded from nodejs.org on Oracle Solaris is not possible due to the proper version of the C++ runtime library not found (the 'GLIBCXX_3.4.20' not found message).
2. Building nodejs/node's master and v4.x from source doesn't work on Oracle Solaris.

2) is tracked with https://github.com/nodejs/node/issues/3179, so please comment in that issue for any problem related to building node on Solaris.

It would be great if we could also solve 1). Let's see what happens on SmartOS.

SmartOS packages for node v4.x require a C++ runtime at version GLIBCXX_3.4.15:

[root@dev ~]# pvs `which node`
        libkstat.so.1 (SUNW_0.7);
        libumem.so.1 (SUNW_1.1);
        libsocket.so.1 (SUNW_1.7);
        libnsl.so.1 (SUNW_1.7, SUNWprivate_1.1);
        librt.so.1 (SUNW_1.4);
        libsendfile.so.1 (SUNW_1.1);
        libstdc++.so.6 (GLIBCXX_3.4.15, CXXABI_1.3);
        libm.so.2 (SUNW_1.2);
        libgcc_s.so.1 (GCC_3.0);
        libpthread.so.1 (SUNW_1.2);
        libc.so.1 (SUNW_1.23, SUNWprivate_1.1);
[root@dev ~]#

This is definitely older than GLIBCXX_3.4.20, so it has higher chances to run on more systems. The node binary links against /opt/local/gcc48/x86_64-sun-solaris2.11/lib/amd64/libstdc++.so.6, and not against /usr/lib/libstdc++.so.6:

[root@dev ~]# ldd `which node`
        libz.so.1 =>     /opt/local/lib/libz.so.1
        libkstat.so.1 =>         /lib/64/libkstat.so.1
        libumem.so.1 =>  /lib/64/libumem.so.1
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        librt.so.1 =>    /lib/64/librt.so.1
        libsendfile.so.1 =>      /lib/64/libsendfile.so.1
        libstdc++.so.6 =>        /opt/local/gcc48/x86_64-sun-solaris2.11/lib/amd64/libstdc++.so.6
        libm.so.2 =>     /lib/64/libm.so.2
        libgcc_s.so.1 =>         /opt/local/gcc48/x86_64-sun-solaris2.11/lib/amd64/libgcc_s.so.1
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libgcc_s.so.1 =>         /opt/local/gcc47/x86_64-sun-solaris2.11/lib/amd64/libgcc_s.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libmd.so.1 =>    /lib/64/libmd.so.1
[root@dev ~]#

And that C++ runtime is compatible with version GLIBCXX_3.4.15:

[root@dev ~]# nm /opt/local/gcc48/x86_64-sun-solaris2.11/lib/libstdc++.so.6.0.19 | grep GLIBCXX | grep 3.4.15
00000000 A GLIBCXX_3.4.15
[root@dev ~]#

I don't know yet what makes the pkgsrc packages on SmartOS link with that specific version of libstdc++ and not the default one, I'd need to investigate further.

And finally, SmartOS pkgsrc packages specifically use GCC 4.8 to build the node binary, whereas it seems that the Node.js release jobs use GCC 4.9.1. This helps make the resulting node binary require an older C++ runtime version.

In other words, it is very likely that the Node.js release jobs could make the resulting node binaries link against an older C++ runtime, which could make these binaries run on Oracle Solaris 11.2 by default.

Just to give more context, at some point a similar issue was encountered for Linux binaries distributed by nodejs.org. To work around this problem, Linux node binaries released on nodejs.org use the RedHat developer toolset to link against a specific C++ runtime that includes a static part with newer C++ symbols so that the dynamic part of the C++ runtime can be found on older system, and the node binary can use newer C++ features by including them directly in the binary.

Unfortunately I don't have a Solaris setup at hand, and that makes testing these theories and possible fixes difficult, if not impossible. But that's a starting point for anyone who would be willing to tackle that problem. I might get some time to work on that in the future, in which case I'll keep you posted.

[alexijs]

@jbergstroem: this should work. However, I believe, it gotta be lower case cc. I relinked them back to solarisstudio compilers (I build apache and php using them and solaris optimization flags without a problem)

[misterdjules]

I don't know yet what makes the pkgsrc packages on SmartOS link with that specific version of libstdc++ and not the default one, I'd need to investigate further.

Actually, just using the GCC 4.8 found in SmartOS pkgsrc packages make the binary link against /opt/local/gcc48/x86_64-sun-solaris2.11/lib/libstdc++.so.6.0.19.

[misterdjules]

@GeorgeBailey Also, the --without-mdb configuration option is not present in node v4.x and later. mdb_v8 is not bundled with node v4.x and later, because bundling the mdb_v8 debugger module in the node binary is actually not useful (it was at some point). Instead, the recommended way to use it is to download the latest release.

[GeorgeBailey]

Have you compiled it successfully via gnu utils?

@alexijs, I was able to compile v0.12, but not v4.2.0.

There are two different categories of problems mentioned in this issue:

@misterdjules, I will start using https://github.com/nodejs/node/issues/3179 for issues building. (your # 1) It makes sense to keep this separate.

So this issue should only be for Solaris 11.2 compatibility with pre-built binaries.

An idea - it would be relevant to know what build environment is used. Is there a wiki on this, so that another person could recreate the build environment on their own?

[misterdjules]

An idea - it would be relevant to know what build environment is used. Is there a wiki on this, so that another person could recreate the build environment on their own?

Definitely. The nodejs/build repository already has some (possibly outdated) documentation. I'm wondering if the best documentation in this case is the source, for instance: https://github.com/nodejs/build/blob/master/setup/smartos/ansible-vars.yaml.

[misterdjules]

@GeorgeBailey @jbergstroem Speaking of https://github.com/nodejs/build/blob/master/setup/smartos/ansible-vars.yaml, I would start investigating replacing gcc49 and gcc49-libs by gcc48 and gcc48-libs and see what impact it has on the C++ runtime that the node binary links to. Chances are that such a binary would work on Solaris 11.2.

[jbergstroem]

@misterdjules: only gcc47 and gcc49 are available in 2014Q3 (which our bots are running). Time to update?

[misterdjules]

@jbergstroem I would suggest testing this with 2014Q4, which is a LTS release that has gcc 4.8. But I'd like to confirm with @jperkin before moving forward to avoid wasting time.

@jperkin IIRC, 2014Q4 doesn't have OpenSSL 1.0.2, but in this case it should not be a problem since node would be built with the bundled OpenSSL in deps/.

[alexijs]

@GeorgeBailey: let me advice you on compatibility with sol v10 and v11: you should compile it on v10 and you may be transfer the binary to the same arch on v11. You wont be able to do it vice versa. However, I would suggest compile it separately for different platforms, make a package, and add it to your internal repos for each os version.

[misterdjules]

@alexijs Thank you very much for sharing your progress, it is very much appreciated. Do you mind continuing that discussion in https://github.com/nodejs/node/issues/3179 though? This issue is about not being able to run binaries available on nodejs.org, and discussing build issues just makes things very confusing.

[misterdjules]

Regarding running v4.x binaries available on nodejs.org on Solaris 11.2, here's what I think is an accurate summary of the current situation:

1. Binaries available on nodejs.org are built using gcc/g++ 4.9 because:
   • The build fails with gcc/g++ 4.7.x.
   • g++ 4.8 is not available on the build machines currently used to build release binaries for sunos.
2. It would be possible to upgrade build machines to a pkgsrc version that provides gcc/g++ 4.8 and build sunos binaries with this toolchain. Unfortunately, that means node would be dynamically linked to a C++ runtime with a path that is not the default path (/opt/local/gcc48/x86_64-sun-solaris2.11/lib/libstdc++.so.6.0.19). Thus, any user who would want to run node on sunos would need to install that dynamic library at the same path by installing the optional gcc-4.8 package on SmartOS. I'm not sure what would be the impact on Solaris 11.2. So that would break even more users than building node binaries with gcc/g++ 4.9.
3. The reason why node v4.x doesn't build with g++ is because of a g++ bug whose fix hasn't been backported to g++ 4.7.x. However, it seems that node v4.x can actually be built and passes all tests with g++ 4.7 with a simple patch.
4. We could also explore other options, such as providing the C++ runtime library in the binary tarball, or having a fully static node binary on sunos, but both these potential solutions seem overkill compared to 3).
5. There's also the option of letting package managers handle these differences, but again, 3) seems non-intrusive enough to be worth trying.

@nodejs/build @nodejs/collaborators As mentioned above, 3) seems like it would not result in any regression, and it could allow more Solaris users to use v4.x binaries from nodejs.org. Thoughts on trying that approach?

[evanlucas]

I think that opening a PR with the patch will be the best way to get actual feedback on whether that patch is acceptable (which doesn't appear to be a major change) would be the best step. Maybe /cc @indutny ?

[misterdjules]

@evanlucas Good point, will do that soon, thank you!

[misterdjules]

@evanlucas Done with #3391.

[alexijs]

@misterdjules wrote: 'g++ 4.8 is not available on the build machines currently used to build release binaries for sunos.' Why are they not available? Its latest supported by Oracle version. May be it makes sense to build Node for Solaris and Smart OS separately for each OS? I think calling a smartOS package a solaris package is too misleading...

Also, many companies release binaries for each OS version separately. I thinks its a very good practice especially from QA standpoint.

In ideal world there would be node-sol-10.pkg, node-sol-11.pkg, and node-smartos.pkg And in the perfect world there would be a node repos for online installation.

[alexijs]

@misterdjules: I moved compilation posts to the topic you suggested.

[misterdjules]

@alexijs Excellent thank you!

@misterdjules wrote: 'g++ 4.8 is not available on the build machines currently used to build release binaries for sunos.' Why are they not available?

Because the build machines use a version of pkgsrc that is tool old to include g++ 4.8.

May be it makes sense to build Node for Solaris and Smart OS separately for each OS? I think calling a smartOS package a solaris package is too misleading...

Indeed, that seems like a good suggestion, probably worth exploring.

[jbergstroem]

I'm happy to update our build toolset to 4.8, I just want to exhaust the discussion in https://github.com/nodejs/node/pull/3391 first.

[jbergstroem]

I'd like to avoid splitting up compilation, mainly because we don't have slaves in our environment covering solaris, but also that it seems to be close enough to find a shared path. If anyone wants to talk to the build team (https://gitter.im/nodejs/build) about contributing room for vm's running solaris now's the time! :+1:

[alexijs]

@jbergstroem: since node getting more and more mature, may be it makes sense to make an official Solaris repo to bring node into enterprises and make life of UNIX geeks easier? I'd be happy to contribute to the best of my abilites :)

[misterdjules]

I'm happy to update our build toolset to 4.8, I just want to exhaust the discussion in #3391 first.

@jbergstroem I changed my mind and I don't think it's a good idea anymore for the reason described in one of my previous comments:

It would be possible to upgrade build machines to a pkgsrc version that provides gcc/g++ 4.8 and build sunos binaries with this toolchain. Unfortunately, that means node would be dynamically linked to a C++ runtime with a path that is not the default path (/opt/local/gcc48/x86_64-sun-solaris2.11/lib/libstdc++.so.6.0.19). Thus, any user who would want to run node on sunos would need to install that dynamic library at the same path by install the optional gcc-4.8 package on SmartOS. I'm not sure what would be the impact on Solaris 11.2. So that would break even more users than building node binaries with gcc/g++ 4.9.

@alexijs' suggestion seems to make more sense to me, also because that would mean that we'd have the infrastructure to test, debug and fix builds for Solaris:

since node getting more and more mature, may be it makes sense to make an official Solaris repo to bring node into enterprises and make life of UNIX geeks easier?

[alexijs]

@misterdjules: this all sound great. However, the main issue here is not the difference in gnu cc versions among SmartOS and Oracle Solaris. The main issue is that build is not compatible with Solaris Studio compiler. Oracle's way of doing things is using Solaris Studio for compilation because due to proprietary optimizations it gives you 20-40% improvement in performance over generic gnu cc. This issue is not related to Node team at all. However, the best way of resolving this issue would be an ability to compile node on Solaris using Solaris Studio. Other open source software can compile using Sun CC fairly easily. Solaris Studio is one of the main selling points for Oracle UNIX. When configured properly it gives you significant performance gain comparing for example to the same build on RedHat or other linux systems. Overwiew http://www.oracle.com/technetwork/server-storage/solarisstudio/overview/index-jsp-142272.html

C/Cpp http://www.oracle.com/technetwork/server-storage/solarisstudio/features/compilers-2332272.html

[GeorgeBailey]

Any ideas what the chance or timeline of an 'official' binary supporting Solaris 10? I wonder if the Solaris Studio compiler would be more compatible with Solaris 10, or if other changes to Node.JS would be necessary...

[alexijs]

@GeorgeBailey: what apps do you run that require Solaris 10? I dont have much experience with this system unfortunately. We had a few boxes running Oracle 10g a while ago there. However, all were moved to 11g on 11.2.

May be it worth trying to build it on Solaris 10 and see if it will work on 11.2. I think it should...However, I dont have any Solaris 10 systems.

[GeorgeBailey]

@alexijs, Let's discuss this on https://github.com/nodejs/node/issues/3408.

[jelmd]

FWIW: Can confirm, that the v5.0.0 binaries from https://nodejs.org/dist/v5.0.0/ do not work on S11.2 and 11.3. Compiled the source using g++ 4.8.2 incl. the following patches: http://iws.cs.uni-magdeburg.de/~elkner/tmp/node5/ (yes, the isnan.patch can be replaced by -D_GLIBCXX_USE_C99_MATH, and no - neither c++03 nor c++11 can be used as a 3rd alternative to fix it). All test succeed except test-tls-ocsp-callback, test-tls-sni-option and test-tls-sni-server-client. Packages are available for testing via http://pkg.cs.ovgu.de/lnf/en/catalog.shtml (IPS) and http://pkg.cs.ovgu.de/LNF/i386/5.11/ (SVR4).

[jbergstroem]

@jelmd thanks for the update. Will revisit next week.

[rvagg]

Could we get some testers on the sunos binaries @ https://nodejs.org/download/nightly/v5.0.1-nightly2015111384bb74547d/ please? I have no idea if we're any closer to rectifying this but it's a new build setup.

[jelmd]

/tmp/node-v5.0.1-nightly2015111384bb74547d-sunos-x64 > bin/node ld.so.1: node: fatal: libc.so.1: version 'ILLUMOS_0.5' not found (required by file bin/node) ld.so.1: node: fatal: libc.so.1: open failed: No such file or directory Killed (S11.3)

[GeorgeBailey]

Same here on Solaris 11.2, both x86 and x64 binaries.

[rvagg]

Sorry, after actually reading some of the content above, particularly the details posted by @misterdjules, I realise this is about solaris vs smartos/illumos and not a smartos version issue which is what I had in my head. The binaries from that nightly would be interesting to test on older versions of smartos but there's nothing in there for vanilla solaris.

[misterdjules]

Here's a work in progress that describes exactly what's going on: https://gist.github.com/misterdjules/eae9ec70dc1d91fb8dd1. Comments and questions welcome. We'll use that document as the basis for improving our SmartOS builds, and also to determine what we can do for Solaris binaries.

[ApolonTorq]

I'm looking at using Node.js on OpenIndiana and/or Solaris 11.x regularly in the future.

The 5.0.1 nightly2015111384bb74547d build of Nodejs resolves the problem for the Hipster version (2015.10 illumos-7fc68dd) but not on the Legacy release (oi_151a8).

Later releases/builds don't appear to include the fix present in the 5.0.1 nightly build referenced. This includes the 6.0.0 series.

How can I help in ensuring Node.js runs "out of the box" in the future on Open Indiana?

[jbergstroem]

The plan (at least my plan, hehe) is to get access to a wider range of sunos derivatives and figure out the lowest common denominator in terms of toolchain that works on a wide range of deployments. I think its safe to say that smartos and sunos could benefit from two sets of releases though. Guessing Openindiana and smartos would play just fine together.

[misterdjules]

figure out the lowest common denominator in terms of toolchain that works on a wide range of deployments

I'm concerned that this approach would work mostly by luck (as it's been the case for current sunos builds for various solaris-derivatives platforms as demonstrated by https://gist.github.com/misterdjules/eae9ec70dc1d91fb8dd1) and would fail unexpectedly sometimes.

In general it seems to me that leaving the work of providing node binaries to packagers would make more sense on most platforms: they know the platform best to choose the most appropriate toolchain and build settings.

[misterdjules]

@ApolonTorq What errors do you get precisely? Do you mind creating a separate issue with a detailed description of the problem (error messages for each version, etc.)?

[ApolonTorq]

@misterdjules The errors are different unresolved shared library references for the different platforms. I'll create separate issues as requested. Note that if we decide to go with the latest Open Indiana hipster version then the shared library issues don't occur for both the current V5.5.0 and the latest nightly builds.

[jbergstroem]

Seeing how this hasn't made much progress as of late I just wanted to give an update. The build group still hasn't gotten access to solaris vm's. If any company and/or provider would step up it would likely make things go considerably faster. If you're interested, ping us over at https://github.com/nodejs/build.

[No9]

So @ptribble has just announced some great work on the solaris build issue https://twitter.com/ptribble/status/728954509146378240 The actual builds are available here http://petertribble.co.uk/Solaris/node.html Would be great if we could utilize some of this work more formally

[ptribble]

Just a commentary.

I've been shipping Solaris 10 packages for almost 5 years; my build machine is, fortunately, not yet at death's door.

On my Solaris 10 builds I ship the matching gcc libs from the compiler I use. (This is essential on Solaris 10 as the shipped gcc is 3.4.3 and completely ancient. But in the general case you simply can't trust the environment where the package is to be deployed to have matching versions.)

Labelling the binary as a SunOS build is incorrect, and leads to incorrect expectations by users.

I wouldn't ever have a reasonable expectation that an illumos built node would run on Solaris 11 (or vice-versa). They have 5 years of divergence. Something from Solaris 10 should run on both, but in this case lacks a few features, and it would be better to build native. Within both the Solaris 11 and illumos branches, you still have to build on something that's a few revisions back to allow binary compatibility to do its job. On the illumos side, a vanilla OmniOS r151014 might be a good base.)

[jbergstroem]

@ptribble: we've inherited the smartos/sunos build from the old nodejs org. I agree that naming it sunos is incorrect but seeing how we don't have any solaris hardware/vm to build on/support we've decided to not make any changes. If we get resources we will likely revisit and rename.

[jelmd]

Not sure, how you build it, but if you do it on an Illumos based Distro, why not name it this way (i.e. -${distro}-). This way one knows, where the binary package definitely should work. For all others it is a hint, may be or may be not ...

[MaheshVelankar]

@ptribble: Peter,

I followed your instructions on this page. http://www.petertribble.co.uk/Solaris/node.html

I tried V7 current version

Successfully bunzip-ed the package and installed it with pkgadd command as you have instructed.

Added /opt/Node/bin to path and tested the following

which node /opt/Node/bin/node

and

which npm /opt/Node/bin/npm

That confirms that PATH is good.

however the following is failing npm -v /usr/bin/env: Invalid argument

Please let me know how to proceed with this.

My solaris is

uname -a SunOS 5.10 Generic_150400-35 sun4v sparc sun4v

Please let me know if I should provide more details.

Thanks -Mahesh

[ptribble]

SunOS 5.10 Generic_150400-35 sun4v sparc sun4v

That's a SPARC system, which isn't supported by Node.JS at all, so that's an entirely different issue.