can't load library xxx.so / can't resolve symbol xxx

[ElTopo]

Hello,

I just found that some executables report "can't load library", for example:

# make
/tmp/mnt/sda1/optware-ng/bin/make: can't load library 'libdl.so.1'
# /opt/bin/busybox
/tmp/mnt/sda1/optware-ng/bin/busybox: can't load library 'librpc-uclibc.so.1'
# zerotier-cli --help
/tmp/mnt/sda1/optware-ng/sbin/zerotier-one: can't load library 'libm.so.1'

and some report "can't resolve symbol", for example:

# m4
/tmp/mnt/sda1/optware-ng/bin/m4: can't resolve symbol '_obstack_begin'

zerotier-cli was built by me and it was working fine.

I checked my system and there's no "libdl.so.1" and "libm.so.1" under /lib and /opt/lib.

The system is updated (ipkg update && ipkg upgrade).

[alllexx88]

Hi,

These are actually four separate issues that you reported. make and m4 fixed with d70f852a927aec671f04083a1108d0ba4cfce1d6 and 710e3bf819b8a773f67aa15c346e389402407392 respectively. busybox-base (and some other packages too) have an additional librpc-uclibc dependency now, which was probably not installed due to -force-depends switch. zerotier-cli just needs to be rebuilt against current dev packages.

These issues in general popped up due to upgrade to uClibc-ng 1.0.26. There're three main changes that break some things:

1. All shared lib functionality is now in the libc.so.0: i.e., there're no libdl.so.1, libpthread.so.1, libm.so.1, etc..
2. uClibc-ng has internal obstack support removed.
3. uClibc-ng has internal RPC support removed.

All these issues have been addressed by me a while ago. I wrote some scripts to scan through built packages to find out which packages have to be rebuilt to:

1. To remove linkage with shared libdl, libpthread, libm, etc..
2. To remove use of uClibc-ng obstack, and use private obstack implementations instead (mlocate package had to be patched to add such an implementation).
3. To use librpc-uclibc library (an openwrt fork of uClibc RPC) for RPC implementation.

All the needed packages have been rebuilt now, and current versions should be fine. The m4 and make problems you've reported is due to some packages packaging issues: their makefiles didn't remove older versions on packaging; hence on the indexing stage multiple versions (the older and the newer) of such packages existed, and the oldest one was getting indexed.

I've now grepped through sources to find most of such packaging issues and fixed them in one go: 340b56e7701f2aacf32b2789750e75d54c93b9a1. This still leaves room for packaging issues that escaped my regex grep, so please create issues on similar errors. Thanks

[ElTopo]

@alllexx88 Thanks for the reply.

Note that I didn't build make/m4/busybox, they are binaries from optware-ng official packages. I will try to rebuild zerotier but I have to wait before make/m4 work correctly.

Just upgraded make/m4 packages. I am working on zerotier now.

[ElTopo]

@alllexx88 I just got another problem with libpthread.

I am still not sure how to use shared libpthread (as I did before), I have this simple mytest.c:

#include <stdio.h>
#include <pthread.h>

int main (int argc, char *argv)
{
        pthread_mutex_t _mh;
        pthread_mutex_init(&_mh, NULL);
        printf("hello\n");
        return 0;
}

I can only build it statically (link with -static) to run it correctly.

On the other hand, if I just use

gcc -o mytest mytest.c

I got:

# ./mytest
/tmp/mnt/sda1/usbhome/lxl/test/mytest: can't resolve symbol 'pthread_mutex_init'
# ldd ./mytest
        libc.so.0 => /lib//libc.so.0 (0x2aac0000)
# lddtree ./mytest
mytest => ./mytest (interpreter => /opt/lib/ld-uClibc.so.0)
    libc.so.0 => /opt/lib/libc.so.0
        ld-uClibc.so.1 => /opt/lib/ld-uClibc.so.1

Can you help me to find out what went wrong?

[alllexx88]

@ElTopo Thanks for the report. The issue was with the dynamic linker /opt/lib/ld-uClibc.so.0. Unless RPATH was specified, it was looking for libs in /lib dir, even though uClibc-ng was configured with /opt prefix. Looks like uClibc-ng bug to me. Fixed with 32a49d49abfd06f692c780ddf9f436be6ff78f2a (and 3cb31372272c1eb0806bee4fa948242c126fae70 as a follow-up). If you're using the armv7 feed, the fixed binaries are online, mipsel and armv5 ones are currently building.

[alllexx88]

P.S. You don't need to link with libpthread now, but doing so (passing -lpthread or -pthread flags) will not cause linking error, since uClibc-ng comes with a dummy static libpthread.a (installed by libc-dev package). Compare:

[root@unknown src]$ cat mytest.c
#include <stdio.h>
#include <pthread.h>

int main (int argc, char *argv)
{
        pthread_mutex_t _mh;
        pthread_mutex_init(&_mh, NULL);
        printf("hello\n");
        return 0;
}
[root@unknown src]$ gcc mytest.c -o mytest
[root@unknown src]$ ./mytest
hello

to

[root@unknown src]$ gcc mytest.c -o mytest -pthread
[root@unknown src]$ ./mytest
hello
[root@unknown src]$ gcc mytest.c -o mytest -lpthread
[root@unknown src]$ ./mytest
hello
[root@unknown src]$ objdump -p mytest|grep NEEDED
  NEEDED               libc.so.0
[root@unknown src]$ ldd ./mytest
        libc.so.0 => /opt/lib//libc.so.0 (0x40208000)
        ld-uClibc.so.1 => /opt/lib/ld-uClibc.so.0 (0x400b2000)

[ElTopo]

@alllexx88 Thanks, it's really helpful!

I am still working on zerotier client.

I have two optware-ng devices:

1. an ASUS router (AC-RT66U), arch: mips (mipsel-ng), with the latest/updated packages, I got this weird error:

# make one
...
...
...
g++ -O3 -fstack-protector -Wall -Wno-unused-result -Wreorder -fPIE -std=c++11 -pthread  -DNDEBUG -D_FORTIFY_SOURCE=2 -DZT_USE_MINIUPNPC -DMINIUPNP_STATICLIB -DMINIUPNPC_SET_SOCKET_TIMEOUT -DMINIUPNPC_GET_SRC_ADDR -D_BSD_SOURCE -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -DOS_STRING=\"Linux\" -DMINIUPNPC_VERSION_STRING=\"2.0\" -DUPNP_VERSION_STRING=\"UPnP/1.1\" -DENABLE_STRNATPMPERR -DZT_NO_TYPE_PUNNING -DZT_BUILD_PLATFORM=1 -DZT_BUILD_ARCHITECTURE=5 -DZT_SOFTWARE_UPDATE_DEFAULT="\"disable\"" -pie -Wl,-z,relro,-z,now -o zerotier-one controller/EmbeddedNetworkController.o controller/JSONDB.o node/C25519.o node/Capability.o node/CertificateOfMembership.o node/CertificateOfOwnership.o node/Cluster.o node/Identity.o node/IncomingPacket.o node/InetAddress.o node/Membership.o node/Multicaster.o node/Network.o node/NetworkConfig.o node/Node.o node/OutboundMulticast.o node/Packet.o node/Path.o node/Peer.o node/Poly1305.o node/Revocation.o node/Salsa20.o node/SelfAwareness.o node/SHA512.o node/Switch.o node/Tag.o node/Topology.o node/Utils.o osdep/ManagedRoute.o osdep/Http.o osdep/OSUtils.o service/ClusterGeoIpService.o service/SoftwareUpdater.o ext/http-parser/http_parser.o osdep/PortMapper.o ext/miniupnpc/connecthostport.o ext/miniupnpc/igd_desc_parse.o ext/miniupnpc/minisoap.o ext/miniupnpc/minissdpc.o ext/miniupnpc/miniupnpc.o ext/miniupnpc/miniwget.o ext/miniupnpc/minixml.o ext/miniupnpc/portlistingparse.o ext/miniupnpc/receivedata.o ext/miniupnpc/upnpcommands.o ext/miniupnpc/upnpdev.o ext/miniupnpc/upnperrors.o ext/miniupnpc/upnpreplyparse.o ext/libnatpmp/natpmp.o ext/libnatpmp/getgateway.o service/OneService.o one.o osdep/LinuxEthernetTap.o
/opt/bin/ld: errno: TLS definition in /tmp/mnt/sda1/optware-ng/bin/../lib/gcc/mipsel-buildroot-linux-uclibc/7.2.0/../../../libc.so.1 section .tbss mismatches non-TLS reference in /tmp/mnt/sda1/optware-ng/bin/../lib/gcc/mipsel-buildroot-linux-uclibc/7.2.0/../../../libpthread.so
/tmp/mnt/sda1/optware-ng/bin/../lib/gcc/mipsel-buildroot-linux-uclibc/7.2.0/../../../libc.so.1: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make-linux-opt.mk:207: recipe for target 'one' failed
make: *** [one] Error 1

# make debug
...
...
...
g++ -Wall -g -O -std=c++11 -pthread  -D_FORTIFY_SOURCE=2 -DZT_USE_MINIUPNPC -DMINIUPNP_STATICLIB -DMINIUPNPC_SET_SOCKET_TIMEOUT -DMINIUPNPC_GET_SRC_ADDR -D_BSD_SOURCE -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -DOS_STRING=\"Linux\" -DMINIUPNPC_VERSION_STRING=\"2.0\" -DUPNP_VERSION_STRING=\"UPnP/1.1\" -DENABLE_STRNATPMPERR -DZT_TRACE -DZT_NO_TYPE_PUNNING -DZT_BUILD_PLATFORM=1 -DZT_BUILD_ARCHITECTURE=5 -DZT_SOFTWARE_UPDATE_DEFAULT="\"disable\""  -o zerotier-one controller/EmbeddedNetworkController.o controller/JSONDB.o node/C25519.o node/Capability.o node/CertificateOfMembership.o node/CertificateOfOwnership.o node/Cluster.o node/Identity.o node/IncomingPacket.o node/InetAddress.o node/Membership.o node/Multicaster.o node/Network.o node/NetworkConfig.o node/Node.o node/OutboundMulticast.o node/Packet.o node/Path.o node/Peer.o node/Poly1305.o node/Revocation.o node/Salsa20.o node/SelfAwareness.o node/SHA512.o node/Switch.o node/Tag.o node/Topology.o node/Utils.o osdep/ManagedRoute.o osdep/Http.o osdep/OSUtils.o service/ClusterGeoIpService.o service/SoftwareUpdater.o ext/http-parser/http_parser.o osdep/PortMapper.o ext/miniupnpc/connecthostport.o ext/miniupnpc/igd_desc_parse.o ext/miniupnpc/minisoap.o ext/miniupnpc/minissdpc.o ext/miniupnpc/miniupnpc.o ext/miniupnpc/miniwget.o ext/miniupnpc/minixml.o ext/miniupnpc/portlistingparse.o ext/miniupnpc/receivedata.o ext/miniupnpc/upnpcommands.o ext/miniupnpc/upnpdev.o ext/miniupnpc/upnperrors.o ext/miniupnpc/upnpreplyparse.o ext/libnatpmp/natpmp.o ext/libnatpmp/getgateway.o service/OneService.o one.o osdep/LinuxEthernetTap.o
/opt/bin/ld: errno: TLS definition in /tmp/mnt/sda1/optware-ng/bin/../lib/gcc/mipsel-buildroot-linux-uclibc/7.2.0/../../../libc.so.1 section .tbss mismatches non-TLS reference in /tmp/mnt/sda1/optware-ng/bin/../lib/gcc/mipsel-buildroot-linux-uclibc/7.2.0/../../../libpthread.so
/tmp/mnt/sda1/optware-ng/bin/../lib/gcc/mipsel-buildroot-linux-uclibc/7.2.0/../../../libc.so.1: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make-linux-opt.mk:207: recipe for target 'one' failed
make[1]: *** [one] Error 1
make[1]: Leaving directory '/tmp/mnt/sda1/usbhome/lxl/GITs/zerotierone'
make-linux-opt.mk:229: recipe for target 'debug' failed
make: *** [debug] Error 2

2. a seagate NAS, arch: armv5tejl(armv5eabi-ng-legacy), also with lastest/updated packages, the exactly same zerotier source compiled OK and I now have a working zerotier client running.

Still looking at code....

Both devices worked before the recent libc-dev/uclibc-ng update.

[alllexx88]

@ElTopo This file that linker complains should not exist:

/tmp/mnt/sda1/optware-ng/bin/../lib/gcc/mipsel-buildroot-linux-uclibc/7.2.0/../../../libpthread.so
#which evaluates to:
/opt/lib/libpthread.so

Are you sure you have latest uclibc-opt and libc-dev? To make perfectly sure, can you run this:

ipkg update
ipkg install -force-reinstall uclibc-opt libc-dev

/opt/lib/libpthread.so was previously being provided by libc-dev as a link to /opt/lib/libpthread.so.1 from uclibc-opt. Neither /opt/lib/libpthread.so.1 in uclibc-opt nor /opt/lib/libpthread.so in libc-dev are present in current package versions. If reinstalling doesn't help, maybe, some debris left? In this case, can you try a fresh bootstrap?

[ElTopo]

@alllexx88 somehow I still have /opt/lib/libpthread.so:

# dir /opt/lib/libpthread.so
lrwxrwxrwx 1 lxl root 20 Oct 23 16:00 /opt/lib/libpthread.so -> /lib/libpthread.so.0*

Even after I did

ipkg update
ipkg install -force-reinstall uclibc-opt libc-dev

My other arm system does not have /opt/lib/libpthread.so.

So I just manually removed /opt/lib/libpthread.so and zerotier now indeed compile OK.

So it seems that the system does have some debris left. I remember I saw some invalid symbolic links left under /opt/lib and I removed them manually.

When you said "try a fresh bootstrap", did you mean running this?

wget -O - http://ipkg.nslu2-linux.org/optware-ng/bootstrap/buildroot-mipsel-ng-bootstrap.sh | sh

or rebuilding the whole /opt directory starting from a totally empty dir?

Thanks bro, you are very helpful to me~

[alllexx88]

# dir /opt/lib/libpthread.so
lrwxrwxrwx 1 lxl root 20 Oct 23 16:00 /opt/lib/libpthread.so -> /lib/libpthread.so.0*

This explains everything 😈 Looks like you created it manually when you discovered that shared libpthread was missing. Stick to a simple rule: don't mix firmware and optware binaries, as they're incompatible. The latter means "avoid use of any libraries other than ones from the feeds or those you compiled yourself" when compiling from source with Optware-ng dev tools.

When you said "try a fresh bootstrap", did you mean running this? ... or rebuilding the whole /opt directory starting from a totally empty dir?

I meant the latter, but I don't think you need to do it.

Thanks bro, you are very helpful to me~

No problem. Glad we got this sorted out 😃

[ElTopo]

@alllexx88 BIG thanks!

this issue is now closed.