"java: symbol lookup error: libjunixsocket-native-2.2.0.so: undefined symbol: stat" on machines with older glibc (< 2.14)

[QuChen88]

This is a continuation of the previously closed issue https://github.com/kohlschutter/junixsocket/issues/69

I was able to reproduce this same error on my local machine. using version 2.2.0. I am using OpenJDK 1.8, Linux 4.9.217, X86_64.

I haven't tried to use the newest version 2.3.1 yet. But I was able to narrow down the exact line of code that caused this error to occur in 2.2.0. It happened for me in AFUNIXServerSocket.bind(). Hope this helps.

[captainpash]

I'm experiencing this issue when building for an EC2 instance.. It doesn't work for https://aws.amazon.com/marketplace/pp/B007O1K7CE?qid=1596056219340&sr=0-9&ref_=srh_res_product_title. It works fine before 2.2. I tried 2.3.1 and that failed as well. Previously I was using 2.0.4.

[QuChen88]

Yes, 2.0.4 didn't have this same problem. I tried it and it worked fine. I guess this means this was introduced from 2.2 version and onwards.

[kohlschuetter]

Please provide the error log using version 2.3.2 from https://github.com/kohlschutter/junixsocket/releases/tag/junixsocket-parent-2.3.2 Thanks!

[captainpash]

This happens during the test part of our gradle build.

> Task :test ../bin/java: symbol lookup error: /tmp/libtmp3306106139925279461libjunixsocket-native-2.2.0.so: undefined symbol: stat

[kohlschuetter]

This is a known error with 2.2.0, and has been fixed in issue #69

If you're getting this error when using junixsocket 2.3.2, then that means you still have an old version 2.2.0 somewhere on your classpath.

[captainpash]

/JDK8/JDK8-1.0.291324.0/AL2012/DEV.STD.PTHREAD/build/jdk1.8/bin/java: symbol lookup error: /tmp/libtmp3417328399583205286libjunixsocket-native-2.3.1.so: undefined symbol: stat

Still seeing it in 2.3.1

[captainpash]

@kohlschuetter sorry see above.

[kuldeep992]

@kohlschuetter I'm facing this issue as well. Details below:

Error message: /java: symbol lookup error: /tmp/libtmp8788978717343818171libjunixsocket-native-2.3.0.so: undefined symbol: stat Operating system: CentOS release 6.10 (Final) Host: Docker container of base CentOS-6 image. (docker pull centos:6) Architecture: x86_64

I've tried it with 2.3.0, 2.3.1, 2.3.2, and faced same issue with all three. Another fact is that the steps and junixsocket libs (2.3.2) are working fine on these docker containers: centos-7, centos-8, ubuntu, debian hosts.

Let me know if you need further information.

[marciocarmona]

It looks like the issue is caused by the glibc version used to link the junixsocket-native libs.

I noticed that it first try to load the /lib/amd64-Linux-clang/jni/libjunixsocket-native-2.3.1.so lib, and throws this error

java.lang.UnsatisfiedLinkError: /tmp/libtmp8995569917288017442libjunixsocket-native-2.3.1.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/libtmp8995569917288017442libjunixsocket-native-2.3.1.so)

Then it tries and successfully loads the /lib/amd64-Linux-clang/jni/libjunixsocket-native-2.3.1.nodeps.so lib, however it fails the call to socket.bind(...) with the reported error:

java: symbol lookup error: libjunixsocket-native-2.2.0.so: undefined symbol: stat

I checked and my env has glibc 2.12 only:

% ldd --version
ldd (GNU libc) 2.12

I also saw that GLIBC_2.14 is only linked for the memcpy call:

% objdump -T /tmp/libtmp9129060092661420511libjunixsocket-native-2.3.1.so

/tmp/libtmp9129060092661420511libjunixsocket-native-2.3.1.so:     file format elf64-x86-64

DYNAMIC SYMBOL TABLE:
0000000000000000  w   D  *UND*  0000000000000000  Base        __gmon_start__
0000000000000000  w   D  *UND*  0000000000000000  Base        _ITM_deregisterTMCloneTable
0000000000000000  w   D  *UND*  0000000000000000  Base        _ITM_registerTMCloneTable
0000000000000000  w   DF *UND*  0000000000000000  GLIBC_2.2.5 __cxa_finalize
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 socket
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strcpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 connect
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strncpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 unlink
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 listen
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 chmod
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 mkstemp
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 calloc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strlen
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memset
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __errno_location
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 bind
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getsockopt
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 shutdown
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 recv
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.14  memcpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 setsockopt
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 malloc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 ioctl
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 sendmsg
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __cmsg_nxthdr
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 rename
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 close
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strerror_r
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 accept
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __xstat
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 recvmsg
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 snprintf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 free
0000000000003920 g    DF .text  0000000000000901  Base        Java_org_newsclub_net_unix_NativeUnixSocket_bind
0000000000003230 g    DF .text  000000000000002d  Base        Java_org_newsclub_net_unix_NativeUnixSocket_initFD
00000000000061c0 g    DF .text  0000000000000013  Base        Java_org_newsclub_net_unix_NativeUnixSocket_maxAddressLength
00000000000054e0 g    DF .text  00000000000001cc  Base        Java_org_newsclub_net_unix_NativeUnixSocket_setSocketOptionInt
00000000000052f0 g    DF .text  00000000000001e4  Base        Java_org_newsclub_net_unix_NativeUnixSocket_getSocketOptionInt
0000000000004600 g    DF .text  0000000000000527  Base        Java_org_newsclub_net_unix_NativeUnixSocket_read
0000000000005a40 g    DF .text  000000000000012b  Base        Java_org_newsclub_net_unix_NativeUnixSocket_setCreated
0000000000005dd0 g    DF .text  000000000000012b  Base        Java_org_newsclub_net_unix_NativeUnixSocket_setCreatedServer
000000000000663c g    DF .fini  0000000000000000  Base        _fini
00000000000051b0 g    DF .text  000000000000006e  Base        Java_org_newsclub_net_unix_NativeUnixSocket_shutdown
0000000000003100 g    DF .text  0000000000000048  Base        Java_org_newsclub_net_unix_NativeUnixSocket_capabilities
00000000000061e0 g    DF .text  000000000000021e  Base        Java_org_newsclub_net_unix_NativeUnixSocket_currentRMISocket
0000000000005a00 g    DF .text  000000000000003d  Base        Java_org_newsclub_net_unix_NativeUnixSocket_initServerImpl
0000000000005740 g    DF .text  0000000000000144  Base        Java_org_newsclub_net_unix_NativeUnixSocket_peerCredentials
0000000000003300 g    DF .text  00000000000002d4  Base        Java_org_newsclub_net_unix_NativeUnixSocket_accept
00000000000050e0 g    DF .text  00000000000000d0  Base        Java_org_newsclub_net_unix_NativeUnixSocket_close
00000000000042f0 g    DF .text  000000000000030b  Base        Java_org_newsclub_net_unix_NativeUnixSocket_connect
00000000000030e0 g    DF .text  000000000000000e  Base        Java_org_newsclub_net_unix_NativeUnixSocket_init
0000000000004280 g    DF .text  0000000000000061  Base        Java_org_newsclub_net_unix_NativeUnixSocket_listen
0000000000005b70 g    DF .text  000000000000012b  Base        Java_org_newsclub_net_unix_NativeUnixSocket_setConnected
0000000000006624 g    DF .init  0000000000000000  Base        _init
0000000000006180 g    DF .text  000000000000003d  Base        Java_org_newsclub_net_unix_NativeUnixSocket_attachCloseable
00000000000030f0 g    DF .text  000000000000000e  Base        Java_org_newsclub_net_unix_NativeUnixSocket_destroy
0000000000003150 g    DF .text  0000000000000027  Base        Java_org_newsclub_net_unix_NativeUnixSocket_getFD
0000000000005ca0 g    DF .text  000000000000012b  Base        Java_org_newsclub_net_unix_NativeUnixSocket_setBound
0000000000005f00 g    DF .text  000000000000012b  Base        Java_org_newsclub_net_unix_NativeUnixSocket_setBoundServer
0000000000006030 g    DF .text  000000000000014c  Base        Java_org_newsclub_net_unix_NativeUnixSocket_setPort
0000000000004c90 g    DF .text  0000000000000445  Base        Java_org_newsclub_net_unix_NativeUnixSocket_write
0000000000005220 g    DF .text  00000000000000d0  Base        convertSocketOptionToNative
00000000000056b0 g    DF .text  0000000000000085  Base        Java_org_newsclub_net_unix_NativeUnixSocket_available

After building GLIBC_2.14 locally and adding it to the LD_LIBRARY_PATH everything worked.

The behavior above is described here:

NOTE: On Linux, the junixsocket library is not always linked against glibc. Some distributions use alternative libc implementations, such as musl-libc (on Alpine Linux, for example). In order to support these environments, the native library loader tries to load the junixsocket version for glibc first, and if that doesn’t work, it falls back to the “.nodeps.so” version, which does not have glibc linked.

If upgrading to GLIBC_2.14 is not an option, what should be the steps to use the .nodeps.so with an older version of it?

[marciocarmona]

Everything worked fine after recompiling the code locally as the memcpy call ended up linked to GLIBC_2.2.5:

% objdump -T target/nar/junixsocket-native-2.3.2-amd64-Linux-clang-jni/lib/amd64-Linux-clang/jni/libjunixsocket-native-2.3.2.so | grep memcpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memcpy

Considering that, and that it doesn't actually need the memcpy from GLIBC_2.14, can we force the linker to always link it with 2.2.5 instead? Like described here: https://stackoverflow.com/a/8862631

[marciocarmona]

I was able to address it by following the steps from https://kohlschutter.github.io/junixsocket/customarch.html to build a custom native JAR using Glibc 2.12 and everything works fine now.

[omerbn]

@kohlschuetter @QuChen88 I found a solution!

it appears that stat() doesn't exist in dynamic libc.so. https://www.redhat.com/archives/pam-list/1999-February/msg00082.html

We need to use inline function. I'll open a PR soon.

[omerbn]

@kohlschuetter @QuChen88 PR: https://github.com/kohlschutter/junixsocket/pull/84

[QuChen88]

@omerbn thanks for the fix.

[kohlschuetter]

PR #84 does not solve the issue.

The problem is that when the binary is built on a newer glibc version (2.14 or newer), it will still not work when you try to run it on glibc 2.12.

Here's a workaround:

On your target platform, compile the below file ("stat.c") as described in the source code and make sure the library is loaded before you call junixsocket code (e.g., via export LD_PRELOAD=/path/to/stat.so).

#include <sys/stat.h>

// Hotfix to support junixsocket 2.3 on
// Linux platforms with glibc older than 2.14
//
// Christian Kohlschuetter, 2021-01-04
//
// compile with:
//   gcc -fPIC -shared -o stat.so stat.c
// or
//   clang -fPIC -shared -o stat.so stat.c
//
// Before invoking java, preload library with:
// export LD_PRELOAD=/path/to/stat.so

int stat(const char * path, struct stat * stat_buf) {
    return __xstat(1 /*_STAT_VER*/, path, stat_buf);
}

In order to fix this automatically without user intervention, a couple more steps are necessary. Given that glibc 2.12 is quite outdated (RHEL/CentOS versions 5 and 6), I don't see this as a top priority, but at least this workaround should get you going.

[kohlschuetter]

I think I've fixed this for good in version 2.3.3.

Please verify that you can run the selftest binary from here:

java -jar junixsocket-selftest-2.3.3-20210106.082329-1-jar-with-dependencies.jar

The last line should say "Selftest PASSED" on all platforms. On Windows, "PASSED WITH ISSUES" is also acceptable.

Alternatively, feel free to point your code to the 2.3.3-SNAPSHOT version from this snapshot repository: https://oss.sonatype.org/content/repositories/snapshots.

In any case, please report back as soon as possible. If all goes well, I can release 2.3.3 soon. Thanks!

[QuChen88]

I just downloaded your jar file and ran the self test locally on my Linux 4.9 X86_64 machine. Getting an error

dev-dsk-quchen-2a-b4e11061 % java -jar junixsocket-selftest-2.3.3-20210106.082329-1-jar-with-dependencies.jar

junixsocket selftest

This program determines whether junixsocket is supported on the current platform. The final line should say whether the selftest passed or failed. If the selftest failed, please visit https://github.com/kohlschutter/junixsocket/issues and file a new bug report with the output below.

selftest version 2.3.3-SNAPSHOT

System properties:

awt.toolkit: sun.awt.X11.XToolkit file.encoding: UTF-8 file.encoding.pkg: sun.io file.separator: / java.awt.graphicsenv: sun.awt.X11GraphicsEnvironment java.awt.printerjob: sun.print.PSPrinterJob java.class.path: junixsocket-selftest-2.3.3-20210106.082329-1-jar-with-dependencies.jar java.class.version: 52.0 java.endorsed.dirs: /local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/endorsed java.ext.dirs: /local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/ext:/usr/java/packages/lib/ext java.home: /local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre java.io.tmpdir: /tmp java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.runtime.name: OpenJDK Runtime Environment java.runtime.version: 1.8.0_232-b09 java.specification.name: Java Platform API Specification java.specification.vendor: Oracle Corporation java.specification.version: 1.8 java.vendor: Oracle Corporation java.vendor.url: http://java.oracle.com/ java.vendor.url.bug: http://bugreport.sun.com/bugreport/ java.version: 1.8.0_232 java.vm.info: mixed mode java.vm.name: OpenJDK 64-Bit Server VM java.vm.specification.name: Java Virtual Machine Specification java.vm.specification.vendor: Oracle Corporation java.vm.specification.version: 1.8 java.vm.vendor: Oracle Corporation java.vm.version: 25.232-b09 line.separator: \n os.arch: amd64 os.name: Linux os.version: 4.9.230-0.1.ac.223.84.332.metal1.x86_64 path.separator: : sun.arch.data.model: 64 sun.boot.class.path: /local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/resources.jar:/local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/rt.jar:/local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/sunrsasign.jar:/local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/jsse.jar:/local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/jce.jar:/local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/charsets.jar:/local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/jfr.jar:/local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/classes sun.boot.library.path: /local/apollo/package/local_1/Linux-2.6c2.5-x86_64/JDK8/JDK8-43721.0-0/jdk1.8/jre/lib/amd64 sun.cpu.endian: little sun.cpu.isalist: sun.io.unicode.encoding: UnicodeLittle sun.java.command: junixsocket-selftest-2.3.3-20210106.082329-1-jar-with-dependencies.jar sun.java.launcher: SUN_STANDARD sun.jnu.encoding: UTF-8 sun.management.compiler: HotSpot 64-Bit Tiered Compilers sun.os.patch.level: unknown user.country: US user.dir: /local/home/quchen user.home: /home/quchen user.language: en user.name: quchen user.timezone:

AFUNIXSocket.isSupported: Exception in thread "main" java.lang.UnsatisfiedLinkError: Could not load native library junixsocket-native for architecture amd64-Linux at org.newsclub.net.unix.NativeLibraryLoader.initCantLoadLibraryError(NativeLibraryLoader.java:274) at org.newsclub.net.unix.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:256) at org.newsclub.net.unix.NativeUnixSocket.(NativeUnixSocket.java:40) at org.newsclub.net.unix.AFUNIXSocket.isSupported(AFUNIXSocket.java:179) at org.newsclub.net.unix.selftest.Selftest.checkSupported(Selftest.java:154) at org.newsclub.net.unix.selftest.Selftest.main(Selftest.java:79) Suppressed: java.lang.Exception: No library specified with -Dorg.newsclub.net.unix.library.override= at org.newsclub.net.unix.NativeLibraryLoader.loadLibraryOverride(NativeLibraryLoader.java:209) at org.newsclub.net.unix.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:232) ... 4 more Suppressed: java.lang.UnsatisfiedLinkError: no junixsocket-native-2.3.3-SNAPSHOT in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at org.newsclub.net.unix.NativeLibraryLoader$StandardLibraryCandidate.load(NativeLibraryLoader.java:112) at org.newsclub.net.unix.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243) ... 4 more

[kohlschuetter]

Thanks @QuChen88! I think this is yet another case that I haven't addressed yet. While we don't currently log the libc version, I think your Linux kernel appears too new to be running the old glibc 2.12. What Linux distribution is this?

Can you please provide the output of bash -c "for f in $(ls {/usr,/}{lib,lib64}/libc.*); do echo $f; $f --version || true; echo; done; env; ldd --version"

Can also please include the output of

cat /etc/os-release
lsb_release -a

[QuChen88]

[~] bash -c "for f in $(ls {/usr,/}{lib,lib64}/libc.*); do echo $f; $f --version || true; echo; done; env; ldd --version" ldd (GNU libc) 2.12 Copyright (C) 2010 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Roland McGrath and Ulrich Drepper.

[~] uname -a Linux 4.9.230-0.1.ac.223.84.332.metal1.x86_64

[QuChen88]

I am running on Amazon Linux - as you can launch an instance using Amazon Linux image from AWS EC2.

[kohlschuetter]

@QuChen88 Please provide me with the exact Amazon Linux version and image name you're using. I see you work at Amazon, can you please share an iso/VirtualBox image of the affected OS?

[kohlschuetter]

@QuChen88 Ah, looks like I pointed you at the wrong selftest jar (it didn't have any libraries other than for macOS).

Please try again with this selftest jar

Of note, I tried a Docker image of Amazon Linux. I went back to 2016.09, and it already has a newer glibc than the one you report. Can you please report the contents of /etc/os-release and /etc/system-release? Thanks!

[kohlschuetter]

Closing due to inactivity. More changes went into junixsocket 2.3.3; please try again and re-open this issue if necessary.