search

Level / leveldown

Pure C++ Node.js LevelDB binding. An abstract-leveldown compliant store.
MIT License
775 stars 177 forks source link

dlopen failed: cannot locate symbol "_Unwind_Resume" #705

Closed christianbundy closed 4 years ago

christianbundy commented 4 years ago

Hi! I'm trying to use leveldown on my Android device via Termux, but I'm bumping into this error. Any chance someone knows what this problem might be? I'm using leveldown 5.4.1 on a Google Pixel XL.

internal/modules/cjs/loader.js:1024
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: dlopen failed: cannot locate symbol "_Unwind_Resume" referenced by "/data/data/com.termux/files/home/.npm-global/lib/node_modules/@fraction/oasis/node_modules/leveldown/prebuilds/android-arm64/node.napi.armv8.node"...
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1024:18)
    at Module.load (internal/modules/cjs/loader.js:813:32)
    at Function.Module._load (internal/modules/cjs/loader.js:725:14)
    at Module.require (internal/modules/cjs/loader.js:850:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at load (/data/data/com.termux/files/home/.npm-global/lib/node_modules/@fraction/oasis/node_modules/leveldown/node_modules/node-gyp-build/index.js:20:10)
    at Object.<anonymous> (/data/data/com.termux/files/home/.npm-global/lib/node_modules/@fraction/oasis/node_modules/leveldown/binding.js:1:43)
    at Module._compile (internal/modules/cjs/loader.js:958:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:994:10)
    at Module.load (internal/modules/cjs/loader.js:813:32)
$ npm version
{
  npm: '6.12.0',
  ares: '1.14.0',
  brotli: '1.0.7',
  cldr: '36.0',
  icu: '65.1',
  llhttp: '1.1.4',
  modules: '79',
  napi: '5',
  nghttp2: '1.39.2',
  node: '13.0.0',
  openssl: '1.1.1d',
  tz: '2019c',
  unicode: '12.1',
  uv: '1.32.0',
  v8: '7.8.279.17-node.14',
  zlib: '1.2.11'
}
christianbundy commented 4 years ago

I put together a PR for ARMv8 support, because I saw this:

error loading sodium bindings: No native build was found for platform=android arch=arm64 runtime=node abi=79 uv=1 armv=8 libc=glibc

But the more I think about it, I'm wondering whether I'm already covered because arch=arm64? I have no idea what I'm doing, to be honest, so https://github.com/prebuild/docker-images/pull/14 might be a total mistake. Thanks for putting up with me. :heart:

vweevers commented 4 years ago

See https://github.com/prebuild/docker-images/pull/14#issuecomment-588635467.

Can you try building from source? It might be a gcc mismatch of some sort.

christianbundy commented 4 years ago

Just built from source and I'm having the same problem, which is surprising.

I was able to clone Level/level (e5f8e61) and npm i && npm t worked fine -- but that's Level 6, so maybe that's different?

$ npm ls leveldown
@fraction/oasis@2.13.1 /home/christianbundy/src/fraction/oasis
└─┬ @fraction/flotilla@4.0.0
  └─┬ ssb-blobs@1.2.2
    └─┬ level@5.0.1
      └── leveldown@5.4.1

Here's the full log:

out.txt

There were some copy and paste problems toward the end, maybe from the escape codes used for colors in various postinstall text, so if anything looks funky that's why. I'm going to try downgrading to the Node.js LTS package and see if that helps at all.

christianbundy commented 4 years ago

WAIT I'm a big dummy and I was running oasis (using the globally installed package with the bug) instead of npm start (which would run the code I literally just compiled). The --build-from-source breaks my sodium-native and sharp, which needs libvips, but otherwise this seems to be running!

vweevers commented 4 years ago

Can you summarize what works and what doesn't?

christianbundy commented 4 years ago

When running npm install --verbose with Leveldown as a dependency, it seems that the node-gyp-build command prints the "dlopen failed" error and then tries compiling from source. If I have all of the build dependencies, it compiles from source, but if I don't have the build dependencies then it leaves me with a Leveldown that crashes with the "dlopen failed" error when I start my project. EDIT: Another way to trigger this error is to ^C during the build from source. The logs don't suggest to me that it's using a prebuild, but maybe I don't know what to look for.

My new confusion is that if I try to npm install --verbose --only=prod in the Leveldown repository, it tries to build from source and then fails when port_posix.h tries to include Snappy and I get "'snappy.h' file not found". This only seems to happen when building from the repository, which gives me the impression that when using Leveldown as a dependency I have a mixture of prebuilds and building from source, but maybe there's something else going on.

Note: in the interests of sanity, I've switched to Node.js LTS (12), but I've also cleaned my npm cache and removed all of my node_modules directories so I don't expect it to bite us in the ass while debugging.

vweevers commented 4 years ago

When running npm install --verbose with Leveldown as a dependency, it seems that the node-gyp-build command prints the "dlopen failed" error and then tries compiling from source. If I have all of the build dependencies, it compiles from source, but if I don't have the build dependencies then it leaves me with a Leveldown that crashes with the "dlopen failed" error when I start my project.

That is as expected: node-gyp-build checks if a suitable prebuild exists and then tries to load it (as a smoke test, if you will). If that fails, it falls back to compiling. But if compiling also fails, then you still only have the (broken) prebuild.

My new confusion is that if I try to npm install --verbose --only=prod in the Leveldown repository, it tries to build from source and then fails when port_posix.h tries to include Snappy and I get "'snappy.h' file not found".

Snappy is a git submodule. You may have to git submodule update --init --recursive.

vweevers commented 4 years ago

As for why the prebuild is broken, let's find out what _Unwind_Resume is for and where it is expected to be found. libgcc source code says this:

/* Resume propagation of an existing exception.  This is used after
   e.g. executing cleanup code, and not to implement rethrowing.  */
extern void LIBGCC2_UNWIND_ATTRIBUTE
_Unwind_Resume (struct _Unwind_Exception *);

We disable exceptions though, so I would not expect _Unwind_Resume to be needed:

https://github.com/Level/leveldown/blob/3e1177dd82de45871c144b1f39ea7b4dfe96e4b1/binding.gyp#L33

Maybe dockcross sets some flag that overrides this? Maybe we need to set additional flags like -fno-unwind-tables? If it turns out we can't do without _Unwind_Resume, then this SO suggests we need to statically link libgcc_eh (which defines _Unwind_Resume). Or maybe it's an NDK mismatch. We build against ANDROID_NDK_REVISION=16b ANDROID_NDK_API=21. I have no idea what the implications are.

@ralphtheninja any clue?

@christianbundy Can you send your leveldown.node compiled from source, or share the output of readelf -a leveldown.node? We can have a look at its symbol table and compare it against the prebuild. Of our current prebuilds, only android-arm and android-arm64 reference _Unwind_Resume:

# readelf -a prebuilds/android-arm64/node.napi.armv8.node | grep _Unwind_Resume
00000006ed90  024c00000402 R_AARCH64_JUMP_SL 0000000000000000 _Unwind_Resume + 0
   588: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _Unwind_Resume

# readelf -a prebuilds/android-arm/node.napi.armv7.node | grep _Unwind_Resume
0005e160  00000316 R_ARM_JUMP_SLOT   0004dec9   _Unwind_Resume
     3: 0004dec9   136 FUNC    GLOBAL DEFAULT   11 _Unwind_Resume
ralphtheninja commented 4 years ago

Maybe dockcross sets some flag that overrides this? __

We could start by checking the verbose output during compile time, e.g. npm i --verbose to see the flags passed on to gcc.

ralphtheninja commented 4 years ago

This line seems to indicate that we explicitly set exceptions for Android

https://github.com/Level/leveldown/blob/3e1177dd82de45871c144b1f39ea7b4dfe96e4b1/deps/leveldb/leveldb.gyp#L170

The leveldown/binding.gyp only applies to the c++ code in leveldown, e.g. binding.cc.

vweevers commented 4 years ago

Oh right, I forgot about that. What's the difference between ccflags (where we have -fexceptions) and cflags! (where we have -fno-exceptions)?

ralphtheninja commented 4 years ago

@#$@#$% knows. We need to do some .gyp digging :man_dancing:

vweevers commented 4 years ago

Output of npx prebuildify-cross -i android-arm64 -t 8.14.0 --napi --strip with loglevel silly:

android-arm64-silly.log

Noteworthy:

Extract of one of the compiled leveldb files:

/usr/aarch64-linux-android/bin/aarch64-linux-android-clang++ 
'-DNODE_GYP_MODULE_NAME=leveldb' 
'-DUSING_UV_SHARED=1' 
'-DUSING_V8_SHARED=1' 
'-DV8_DEPRECATION_WARNINGS=1' 
'-D_LARGEFILE_SOURCE' 
'-D_FILE_OFFSET_BITS=64' 
'-DSNAPPY=1' 
'-DLEVELDB_PLATFORM_POSIX=1' 
'-DOS_ANDROID=1' 
'-D_REENTRANT=1' 
'-D_GLIBCXX_USE_C99_MATH' 
[.. linker flags ..]
-fPIC -Wall -Wextra -Wno-unused-parameter -std=c++0x 
-Wno-sign-compare -fPIC -O3 -fno-omit-frame-pointer 
-fno-rtti -std=gnu++0x -MMD -MF 
./Release/.deps/Release/obj.target/leveldb/deps/leveldb/leveldb-1.20/db/builder.o.d.raw   
-c -o Release/obj.target/leveldb/deps/leveldb/leveldb-1.20/db/builder.o 
../deps/leveldb/leveldb-1.20/db/builder.cc
vweevers commented 4 years ago

What's the difference between ccflags (where we have -fexceptions) and cflags! (where we have -fno-exceptions)?

The "!" means: exclude -fno-exceptions from cflags. So a correction: on both our binding.cc and leveldb files, exceptions are enabled.

ralphtheninja commented 4 years ago

So maybe then we should take care to link with whatever is needed.

vweevers commented 4 years ago

And add -fvisibility=hidden while we're at it. If nothing else, that'll reduce binary size a little bit.

vweevers commented 4 years ago

I'd like to explore the option of disabling exceptions for all platforms. As far as I can tell we don't need them (anymore?).

christianbundy commented 4 years ago

That is as expected

Sweet. I assumed that it would either use a prebuild or compile from source, but didn't realize there was a smoke test going on.

Snappy is a git submodule.

Love it, thanks. I should've kept scrolling.

Can you send your leveldown.node compiled from source, or share the output of readelf -a leveldown.node?

On it. Thanks for the instructions!

christianbundy commented 4 years ago

readelf-a.txt

leveldown.node.zip

vweevers commented 4 years ago

Here's the difference in the symbol tables between your build (A) and the prebuild (B).

In A, the _Unwind_Resume symbol has an address. Does this suggest the function is inlined? Maybe we can find out with gdb.

In B, the _Unwind_Resume symbol has no address. The dynamic linker will look for _Unwind_Resume in a shared library. The error that you get suggests there is no such library.

Relocation section '.rela.plt' (Procedure Linkage Table):

Offset        Info         Type              Sym. Value       Sym. Name + Addend
000000063af8  00e700000402 R_AARCH64_JUMP_SL 000000000004ec0c _Unwind_Resume + 0 (A)
00000006ead8  024b00000402 R_AARCH64_JUMP_SL 0000000000000000 _Unwind_Resume + 0 (B)
Symbol table '.dynsym':

Num: Value             Size Type    Bind   Vis      Ndx Name
231: 000000000004ec0c   252 FUNC    GLOBAL DEFAULT   10 _Unwind_Resume (A)
587: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _Unwind_Resume (B)
vweevers commented 4 years ago

@christianbundy For further clues, can you provide the output of $CC -v and env?

vweevers commented 4 years ago

Maybe at this point we should just test a few different prebuilds. One without exceptions, one with a statically linked libgcc_eh.

christianbundy commented 4 years ago 
$ cc -v
clang version 9.0.1
Target: aarch64-unknown-linux-android
Thread model: posix
InstalledDir: /data/data/com.termux/files/usr/bin
$ env
SHELL=/data/data/com.termux/files/usr/bin/bash
PREFIX=/data/data/com.termux/files/usr
PWD=/data/data/com.termux/files/home
EXTERNAL_STORAGE=/sdcard
LD_PRELOAD=/data/data/com.termux/files/usr/lib/libtermux-exec.so
HOME=/data/data/com.termux/files/home
LANG=en_US.UTF-8
ANDROID_RUNTIME_ROOT=/apex/com.android.runtime
TMPDIR=/data/data/com.termux/files/usr/tmp
ANDROID_DATA=/data
TERM=xterm-256color
SHLVL=1
ANDROID_ROOT=/system
BOOTCLASSPATH=/apex/com.android.runtime/javalib/core-oj.jar:/apex/com.android.runtime/javalib/core-libart.jar:/apex/com.android.runtime/javalib/okhttp.jar:/apex/com.android.runtime/javalib/bouncycastle.jar:/apex/com.android.runtime/javalib/apache-xml.jar:/system/framework/framework.jar:/system/framework/ext.jar:/system/framework/telephony-common.jar:/system/framework/voip-common.jar:/system/framework/ims-common.jar:/system/framework/android.test.base.jar:/apex/com.android.conscrypt/javalib/conscrypt.jar:/apex/com.android.media/javalib/updatable-media.jar
ANDROID_TZDATA_ROOT=/apex/com.android.tzdata
PATH=/data/data/com.termux/files/home/.npm-global/bin:/data/data/com.termux/files/usr/bin:/data/data/com.termux/files/usr/bin/applets
_=/data/data/com.termux/files/usr/bin/env
vweevers commented 4 years ago

I installed termux just now, with nodejs-lts, leveldown@5.4.1, on Android 9, ARM 64. Unfortunately I'm unable to reproduce the issue. I tried out gdb and lldb too - to see what our debugging options are - but they both crash (unrelated to leveldown).

christianbundy commented 4 years ago

Hmm, interesting. Maybe try pkg install nodejs to see if the problem is happening on Node 13? I was getting "Segmentation fault" on Node 12 and re-upgraded to Node 13. No core dump or anything useful, so the segfault could've been coming from anywhere.

vweevers commented 4 years ago

(termux is awesome, btw. first time i tried it)

vweevers commented 4 years ago

Same with node 13. So..

Maybe at this point we should just test a few different prebuilds. One without exceptions, one with a statically linked libgcc_eh.

@christianbundy would you trust a binary I'd send, or do you want to compile it yourself?

christianbundy commented 4 years ago

I'd trust a binary, but if it's easier for you I can also compile from source. Makes no difference to me.

vweevers commented 4 years ago

A build without exceptions (compiled on prebuild/android-arm64): node.napi.armv8.tar.gz

To test, run the following commands in an empty directory:

npm i leveldown --ignore-scripts --no-save
curl -OL https://github.com/Level/leveldown/files/4238293/node.napi.armv8.tar.gz
tar -zxvf node.napi.armv8.tar.gz

# Make sure prebuilds are not loaded
rm -rf node_modules/leveldown/prebuilds

mkdir -p node_modules/leveldown/build/Release
mv node.napi.armv8.node node_modules/leveldown/build/Release/leveldown.node
node -e "require('leveldown')('db').open(console.log)"
christianbundy commented 4 years ago 
$ npm i leveldown --ignore-scripts --no-save
npm WARN npm npm does not support Node.js v13.0.0
npm WARN You should probably upgrade to a newer version of node as we
npm WARN can't make any promises that npm will work with this version.
npm WARN Supported releases of Node.js are the latest release of 6, 8, 9, 10, 11, 12.
npm WARN You can find the latest version at https://nodejs.org/
npm WARN enoent ENOENT: no such file or directory, open '/data/data/com.termux/files/home/level-test/package.json'
npm WARN level-test No description
npm WARN level-test No repository field.
npm WARN level-test No README data
npm WARN level-test No license field.

+ leveldown@5.4.1
added 7 packages from 5 contributors in 3.16s
found 0 vulnerabilities

$ curl -OL https://github.com/Level/leveldown/files/4238293/node.napi.armv8.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
   0     0    0     0    0     0      0      0 --:-100  147k  100  147k    0     0  82810      0  0:00:01  0:00:01 --:--:--  216k
$ tar -zxvf node.napi.armv8.tar.gz
node.napi.armv8.node
$
$ # Make sure prebuilds are not loaded
$ rm -rf node_modules/leveldown/prebuilds
$
$ mkdir -p node_modules/leveldown/build/Release
$ mv node.napi.armv8.node node_modules/leveldown/build/Release/leveldown.node
$ node -e "require('leveldown')('db').open(console.log)"

No output. That's good, right? :upside_down_face:

vweevers commented 4 years ago

try ls, there should be a db

christianbundy commented 4 years ago 
$ ls
db  node.api.armv8.tar.gz  node_modules

!!! You did it. Thanks so much for all the work you've put into this issue.

ralphtheninja commented 4 years ago

I'd like to explore the option of disabling exceptions for all platforms. As far as I can tell we don't need them (anymore?).

Aye, I don't see why they should be enabled on Android but not on other platforms.

christianbundy commented 4 years ago

Thanks again!