build: OSX fails to load NIF as symbol _LZ4_compressBound is not found.

[dch]

I'm guessing this is some sort of missing library reference in the build tools, but I am not clear what to look for or fix specifically, sorry.

Symbol not found: _LZ4_compressBound\n  Referenced from: /Volumes/ramdisk/erocksdb/priv/erocksdb.s

full log:

Last login: Tue Jul 21 12:17:37 on ttys000
Darwin: akai

Agent pid 41899
dch 🍺  /projects ❯❯❯j erocks
/repos/erocksdb
dch 🍺  /r/erocksdb git:develop ❯❯❯make test
./rebar compile
==> erocksdb (compile)
/Library/Developer/CommandLineTools/usr/bin/make  all-am
make[2]: Nothing to be done for `all-am'.
test -z "/repos/erocksdb/c_src/system/lib" || ./install-sh -c -d "/repos/erocksdb/c_src/system/lib"
 /bin/sh ./libtool   --mode=install /usr/bin/install -c   libsnappy.la '/repos/erocksdb/c_src/system/lib'
libtool: install: /usr/bin/install -c .libs/libsnappy.1.dylib /repos/erocksdb/c_src/system/lib/libsnappy.1.dylib
libtool: install: (cd /repos/erocksdb/c_src/system/lib && { ln -s -f libsnappy.1.dylib libsnappy.dylib || { rm -f libsnappy.dylib && ln -s libsnappy.1.dylib libsnappy.dylib; }; })
libtool: install: /usr/bin/install -c .libs/libsnappy.lai /repos/erocksdb/c_src/system/lib/libsnappy.la
libtool: install: /usr/bin/install -c .libs/libsnappy.a /repos/erocksdb/c_src/system/lib/libsnappy.a
libtool: install: chmod 644 /repos/erocksdb/c_src/system/lib/libsnappy.a
libtool: install: ranlib /repos/erocksdb/c_src/system/lib/libsnappy.a
test -z "/repos/erocksdb/c_src/system/share/doc/snappy" || ./install-sh -c -d "/repos/erocksdb/c_src/system/share/doc/snappy"
 /usr/bin/install -c -m 644 ChangeLog COPYING INSTALL NEWS README format_description.txt framing_format.txt '/repos/erocksdb/c_src/system/share/doc/snappy'
test -z "/repos/erocksdb/c_src/system/include" || ./install-sh -c -d "/repos/erocksdb/c_src/system/include"
 /usr/bin/install -c -m 644 snappy.h snappy-sinksource.h snappy-stubs-public.h snappy-c.h '/repos/erocksdb/c_src/system/include'
Cloning into 'rocksdb'...
Note: checking out 'rocksdb-3.11.2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at add2ac5... Bump the version to 3.11.2
  GEN      util/build_version.cc
  GEN      util/build_version.cc
  CC       db/builder.o
  CC       db/c.o
  CC       db/column_family.o
  CC       db/compaction.o
  CC       db/compaction_job.o
  CC       db/compaction_picker.o
  CC       db/db_filesnapshot.o
  CC       db/dbformat.o
  CC       db/db_impl.o
  CC       db/db_impl_debug.o
  CC       db/db_impl_readonly.o
  CC       db/db_impl_experimental.o
  CC       db/db_iter.o
  CC       db/experimental.o
  CC       db/event_logger_helpers.o
  CC       db/file_indexer.o
  CC       db/filename.o
  CC       db/flush_job.o
  CC       db/flush_scheduler.o
  CC       db/forward_iterator.o
  CC       db/internal_stats.o
  CC       db/log_reader.o
  CC       db/log_writer.o
  CC       db/managed_iterator.o
  CC       db/memtable_allocator.o
  CC       db/memtable.o
  CC       db/memtable_list.o
  CC       db/merge_helper.o
  CC       db/merge_operator.o
  CC       db/repair.o
  CC       db/slice.o
  CC       db/table_cache.o
  CC       db/table_properties_collector.o
  CC       db/transaction_log_impl.o
  CC       db/version_builder.o
  CC       db/version_edit.o
  CC       db/version_set.o
  CC       db/wal_manager.o
  CC       db/write_batch.o
  CC       db/write_batch_base.o
  CC       db/write_controller.o
  CC       db/write_thread.o
  CC       port/stack_trace.o
  CC       port/port_posix.o
  CC       table/adaptive_table_factory.o
  CC       table/block_based_filter_block.o
  CC       table/block_based_table_builder.o
  CC       table/block_based_table_factory.o
  CC       table/block_based_table_reader.o
  CC       table/block_builder.o
  CC       table/block.o
  CC       table/block_hash_index.o
  CC       table/block_prefix_index.o
  CC       table/bloom_block.o
  CC       table/cuckoo_table_builder.o
  CC       table/cuckoo_table_factory.o
  CC       table/cuckoo_table_reader.o
  CC       table/flush_block_policy.o
  CC       table/format.o
  CC       table/full_filter_block.o
  CC       table/get_context.o
  CC       table/iterator.o
  CC       table/merger.o
  CC       table/meta_blocks.o
  CC       table/plain_table_builder.o
  CC       table/plain_table_factory.o
  CC       table/plain_table_index.o
  CC       table/plain_table_key_coding.o
  CC       table/plain_table_reader.o
  CC       table/table_properties.o
  CC       table/two_level_iterator.o
  CC       util/arena.o
  CC       util/auto_roll_logger.o
  CC       util/bloom.o
  CC       util/build_version.o
  CC       util/cache.o
  CC       util/coding.o
  CC       util/comparator.o
  CC       util/crc32c.o
  CC       util/db_info_dumper.o
  CC       util/dynamic_bloom.o
  CC       util/env.o
  CC       util/env_hdfs.o
  CC       util/env_posix.o
  CC       util/file_util.o
  CC       util/filter_policy.o
  CC       util/hash.o
  CC       util/hash_cuckoo_rep.o
  CC       util/hash_linklist_rep.o
  CC       util/hash_skiplist_rep.o
  CC       util/histogram.o
  CC       util/instrumented_mutex.o
  CC       util/iostats_context.o
  CC       utilities/backupable/backupable_db.o
  CC       utilities/convenience/convenience.o
  CC       utilities/checkpoint/checkpoint.o
  CC       utilities/compacted_db/compacted_db_impl.o
  CC       utilities/document/document_db.o
  CC       utilities/document/json_document_builder.o
  CC       utilities/document/json_document.o
  CC       utilities/flashcache/flashcache.o
  CC       utilities/geodb/geodb_impl.o
  CC       utilities/leveldb_options/leveldb_options.o
  CC       utilities/merge_operators/put.o
  CC       utilities/merge_operators/string_append/stringappend2.o
  CC       utilities/merge_operators/string_append/stringappend.o
  CC       utilities/merge_operators/uint64add.o
  CC       utilities/redis/redis_lists.o
  CC       utilities/spatialdb/spatial_db.o
  CC       utilities/ttl/db_ttl_impl.o
  CC       utilities/write_batch_with_index/write_batch_with_index.o
  CC       utilities/write_batch_with_index/write_batch_with_index_internal.o
  CC       util/event_logger.o
  CC       util/ldb_cmd.o
  CC       util/ldb_tool.o
  CC       util/log_buffer.o
  CC       util/logging.o
  CC       util/memenv.o
  CC       util/murmurhash.o
  CC       util/mutable_cf_options.o
  CC       util/options_builder.o
  CC       util/options.o
  CC       util/options_helper.o
  CC       util/perf_context.o
  CC       util/rate_limiter.o
  CC       util/skiplistrep.o
  CC       util/slice.o
  CC       util/sst_dump_tool.o
  CC       util/statistics.o
  CC       util/status.o
  CC       util/string_util.o
  CC       util/sync_point.o
  CC       util/thread_local.o
  CC       util/thread_status_impl.o
  CC       util/thread_status_updater.o
  CC       util/thread_status_updater_debug.o
  CC       util/thread_status_util.o
  CC       util/thread_status_util_debug.o
  CC       util/vectorrep.o
  CC       util/xfunc.o
  CC       util/xxhash.o
  AR       librocksdb.a
ar: creating archive librocksdb.a
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: librocksdb.a(sync_point.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: librocksdb.a(xfunc.o) has no symbols
Compiled src/erocksdb_bump.erl
Compiled src/erocksdb.erl
Compiling c_src/erocksdb.cc
Compiling c_src/refobjects.cc
Compiling c_src/threading.cc
Compiling c_src/workitems.cc
./rebar eunit
==> erocksdb (eunit)
Compiled test/basho_bench_driver_eldb.erl
Compiled test/cleanup.erl
test/rand_gen_1.erl:33: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
test/rand_gen_1.erl:64: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled test/rand_gen_1.erl
Compiled test/basho_bench_driver_rocksdb.erl
Compiled src/erocksdb_bump.erl
Compiled test/iterators.erl
Compiled test/cacheleak.erl
Compiled src/erocksdb.erl
======================== EUnit ========================
undefined
*** test module not found ***
**erocksdb

=======================================================
  Failed: 0.  Skipped: 0.  Passed: 0.
One or more tests were cancelled.

=WARNING REPORT==== 21-Jul-2015::12:27:39 ===
The on_load function for module erocksdb returned {error,
                                                   {load_failed,
                                                    "Failed to load NIF library: 'dlopen(/repos/erocksdb/priv/erocksdb.so, 2): Symbol not found: _LZ4_compressBound\n  Referenced from: /repos/erocksdb/priv/erocksdb.so\n  Expected in: flat namespace\n in /repos/erocksdb/priv/erocksdb.so'"}}
ERROR: One or more eunit tests failed.
make: *** [test] Error 1
dch 🍺  /r/erocksdb git:develop ❯❯❯                                                  ⏎

[dch]

OSX: Yosemite 10.10.4 (14E46)

uname:

Darwin akai.skunkwerks.at 14.4.0 Darwin Kernel Version 14.4.0
Thu May 28 11:35:04 PDT 2015
root:xnu-2782.30.5~1/RELEASE_X86_64 x86_64

[yosukehara]

Thank you for your report. I've briefly check that as follows:

• https://gist.github.com/yosukehara/2f47306c46ede689441f

It was working fine on my environment. You could check your environment, again.

[mocchira]

@dch missing symbol _LZ4_compressBound is only referenced if you have installed lz4 in your system and visible at compile time.

The below code block is where deciding whether lz4 is linked or not. https://github.com/facebook/rocksdb/blob/rocksdb-3.11.2/build_tools/build_detect_platform#L259-L269

So there are two solutions.

1. Visible lz4 at execution time by setting LD_LIBRARY_PATH and so on.
2. Remove the above code block I mentioned and rebuild

[yosukehara]

I've reproduced this issue as below. I'll check and fix that on my environment then share the report.

$ brew install lz4
==> Downloading https://homebrew.bintray.com/bottles/lz4-r131.yosemite.bottle.tar.gz
######################################################################## 100.0%
==> Pouring lz4-r131.yosemite.bottle.tar.gz
🍺  /usr/local/Cellar/lz4/r131: 12 files, 436K

$ which lz4
/usr/local/bin/lz4

$ ls -l /usr/local/lib/ | grep liblz4*
lrwxr-xr-x   1 yosuke.hara  admin    41 Jul 22 09:46 liblz4.1.7.1.dylib -> ../Cellar/lz4/r131/lib/liblz4.1.7.1.dylib
lrwxr-xr-x   1 yosuke.hara  admin    37 Jul 22 09:46 liblz4.1.dylib -> ../Cellar/lz4/r131/lib/liblz4.1.dylib
lrwxr-xr-x   1 yosuke.hara  admin    31 Jul 22 09:46 liblz4.a -> ../Cellar/lz4/r131/lib/liblz4.a
lrwxr-xr-x   1 yosuke.hara  admin    35 Jul 22 09:46 liblz4.dylib -> ../Cellar/lz4/r131/lib/liblz4.dylib
lrwxr-xr-x   1 yosuke.hara  admin    38 Jun 16 15:18 liblzma.5.dylib -> ../Cellar/xz/5.2.1/lib/liblzma.5.dylib
lrwxr-xr-x   1 yosuke.hara  admin    32 Jun 16 15:18 liblzma.a -> ../Cellar/xz/5.2.1/lib/liblzma.a
lrwxr-xr-x   1 yosuke.hara  admin    36 Jun 16 15:18 liblzma.dylib -> ../Cellar/xz/5.2.1/lib/liblzma.dylib

$ make test
.
.
.
==> erocksdb (eunit)
======================== EUnit ========================
module 'rand_gen_1'
iterators: prev_test (module 'iterators')...
=ERROR REPORT==== 22-Jul-2015::10:05:54 ===
The on_load function for module erocksdb returned {error,
                                                   {load_failed,
                                                    "Failed to load NIF library: 'dlopen(/Users/yosuke.hara/dev/leo-project/test/erocksdb/priv/erocksdb.so, 2): Symbol not found: _LZ4_compressBound\n  Referenced from: /Users/yosuke.hara/dev/leo-project/test/erocksdb/priv/erocksdb.so\n  Expected in: flat namespace\n in /Users/yosuke.hara/dev/leo-project/test/erocksdb/priv/erocksdb.so'"}}

[mocchira]

@dch if you have installed lz4 by package, you might get an old one which version is less than expected by rocksdb. In that case && you want to use lz4, you need to get the new one at https://github.com/Cyan4973/lz4, install and make it visible prior to the old one.

[dch]

thanks all for help @mocchira & @yosukehara t

1. brew rm lz4: erocksdb works fine
2. brew install lz4: erocksdb does not work against lz4 r131
3. I added head "https://github.com/Cyan4973/lz4.git" to lz4.rb recipe, then brew install -v --HEAD lz4 to build from git master: erocksdb does not work.
4. with below patch & lz4 r131 installed via brew: erocksdb does work.

commit a05183d4e300b9d348624e8ac2785430ccdf795a
Author: Dave Cottlehuber <dch@skunkwerks.at>
Date:   Wed Jul 22 16:24:19 2015 +0200

    hack osx

diff --git a/rebar.config b/rebar.config
index b83ff9f..c14b7fc 100644
--- a/rebar.config
+++ b/rebar.config
@@ -6,8 +6,8 @@
         %% Make sure to set -fPIC when compiling rocksdb
              {"CFLAGS", "$CFLAGS -Wall -O3 -fPIC"},
              {"CXXFLAGS", "$CXXFLAGS -std=c++11 -Wall -O3 -fPIC"},
-             {"DRV_CFLAGS", "$DRV_CFLAGS -O3 -Wall -I c_src/rocksdb -I c_src/rocksdb/include"},
-             {"DRV_LDFLAGS", "$DRV_LDFLAGS c_src/rocksdb/librocksdb.a c_src/system/lib/libsnappy.a -lstdc++ -lbz2"}
+             {"DRV_CFLAGS", "$DRV_CFLAGS -O3 -Wall -I c_src/rocksdb -I c_src/rocksdb/include -I /usr/local/include"},
+             {"DRV_LDFLAGS", "$DRV_LDFLAGS c_src/rocksdb/librocksdb.a c_src/system/lib/libsnappy.a -lstdc++ -lbz2 -llz4"}
              ]}.

 {pre_hooks, [{'get-deps', "c_src/build_deps.sh get-deps"},

There may well be a better way to propagate the changes from rocksdb's mk_config.mk which is detected as follows:

CC=cc
CXX=c++
PLATFORM=OS_MACOSX
PLATFORM_LDFLAGS= -lsnappy -lgflags -lz -lbz2 -llz4
JAVA_LDFLAGS= -lsnappy -lz -lbz2 -llz4
VALGRIND_VER=
PLATFORM_CCFLAGS= -DROCKSDB_PLATFORM_POSIX  -Wall -O3 -fPIC -I /ramdisk/erocksdb/c_src/system/include -DOS_MACOSX -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -DLZ4 -Wshorten-64-to-32 -march=native
PLATFORM_CXXFLAGS=-std=c++11 -fPIC -DROCKSDB_PLATFORM_POSIX  -Wall -O3 -fPIC -I /ramdisk/erocksdb/c_src/system/include -DOS_MACOSX -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -DLZ4 -Wshorten-64-to-32 -march=native
PLATFORM_SHARED_CFLAGS=-fPIC
PLATFORM_SHARED_EXT=dylib
PLATFORM_SHARED_LDFLAGS=-dynamiclib -install_name
PLATFORM_SHARED_VERSIONED=true
EXEC_LDFLAGS=
JEMALLOC_INCLUDE=
JEMALLOC_LIB=
ROCKSDB_MAJOR=3
ROCKSDB_MINOR=11
ROCKSDB_PATCH=2
CLANG_SCAN_BUILD=scan-build
CLANG_ANALYZER=/usr/bin/clang++

cc @licenser maybe you can try something similar on SmartOS?

[yosukehara]

@dch Thank you for sharing the log and your patch is informative. Yesterday, I faced the same situation:

1. brew install lz4: erocksdb does not work against lz4 r131
2. I added head "https://github.com/Cyan4973/lz4.git" to lz4.rb recipe, then brew install -v --HEAD lz4 to build from git master: erocksdb does not work.

We'll consider how to fix this issue.

[dch]

I've tried to find a way to detect conditionally if lz4 is available, but as this is OSX we can't guarantee that e.g. pkg-config is installed, & .pc files are available (as they are via homebrew installed lz4). Hopefully somebody else finds a better way!

[mocchira]

@dch Now I'm implementing a conditional build process based on your info build_config.mk which seems pretty reliable and portable. Many thanks!

[mocchira]

@dch Done. https://github.com/leo-project/erocksdb/commit/b7dede0f02a9e1f53658995393517bff5444ad71

Please try it out in your spare time.

[yosukehara]

@mocchira It was working fine w/the latest version on my environment.

$ ls -l /usr/local/lib/ | grep liblz4
-rwxr-xr-x   1 root         admin               84480 Jul 24 13:17 liblz4.1.7.1.dylib
lrwxr-xr-x   1 root         INTRA\Domain Users     18 Jul 24 13:17 liblz4.1.dylib -> liblz4.1.7.1.dylib
-rw-r--r--   1 root         admin               93856 Jul 24 13:17 liblz4.a
lrwxr-xr-x   1 root         INTRA\Domain Users     18 Jul 24 13:17 liblz4.dylib -> liblz4.1.7.1.dylib

$ make test
./c_src/build_deps.sh
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
make[2]: Nothing to be done for `all-am'.
test -z "/Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/lib" || ./install-sh -c -d "/Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/lib"
 /bin/sh ./libtool   --mode=install /usr/bin/install -c   libsnappy.la '/Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/lib'
libtool: install: /usr/bin/install -c .libs/libsnappy.1.dylib /Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/lib/libsnappy.1.dylib
libtool: install: (cd /Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/lib && { ln -s -f libsnappy.1.dylib libsnappy.dylib || { rm -f libsnappy.dylib && ln -s libsnappy.1.dylib libsnappy.dylib; }; })
libtool: install: /usr/bin/install -c .libs/libsnappy.lai /Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/lib/libsnappy.la
libtool: install: /usr/bin/install -c .libs/libsnappy.a /Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/lib/libsnappy.a
libtool: install: chmod 644 /Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/lib/libsnappy.a
libtool: install: ranlib /Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/lib/libsnappy.a
test -z "/Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/share/doc/snappy" || ./install-sh -c -d "/Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/share/doc/snappy"
 /usr/bin/install -c -m 644 ChangeLog COPYING INSTALL NEWS README format_description.txt framing_format.txt '/Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/share/doc/snappy'
test -z "/Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/include" || ./install-sh -c -d "/Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/include"
 /usr/bin/install -c -m 644 snappy.h snappy-sinksource.h snappy-stubs-public.h snappy-c.h '/Users/yosuke.hara/dev/leo-project/leofs/deps/erocksdb/c_src/system/include'
PLATFORM_LDFLAGS="`cat c_src/rocksdb/build_config.mk |grep PLATFORM_LDFLAGS| awk -F= '{print $2}'`" ./rebar compile
cat: c_src/rocksdb/build_config.mk: No such file or directory
==> erocksdb (compile)
./rebar eunit
==> erocksdb (eunit)
======================== EUnit ========================
module 'rand_gen_1'
iterators: prev_test (module 'iterators')...[0.077 s] ok
module 'erocksdb_bump'
module 'erocksdb'
  erocksdb: open_test...ok
  erocksdb: fold_test...ok
  erocksdb: fold_keys_test...ok
  erocksdb: destroy_test...ok
  erocksdb: compression_test...ok
  erocksdb: close_test...ok
  erocksdb: close_fold_test...ok
  [done in 0.021 s]
module 'cleanup'
  cleanup: assumption_test...assumption_test: top
assumption_test: bottom
[0.507 s] ok
  cleanup: open_close_test...[1.017 s] ok
  cleanup: open_exit_test...[1.524 s] ok
  cleanup: iterator_test...[1.519 s] ok
  cleanup: iterator_db_close_test...[2.040 s] ok
  cleanup: iterator_exit_test...[1.529 s] ok
  [done in 8.153 s]
cacheleak: cacheleak_test_ (module 'cacheleak')...RSS1: 191864
RSS1: 197820
RSS1: 200372
RSS1: 200844
RSS1: 203408
RSS1: 201580
RSS1: 201464
RSS1: 201464
RSS1: 201464
RSS1: 203644
[12.254 s] ok
module 'basho_bench_driver_rocksdb'
module 'basho_bench_driver_eldb'
=======================================================
  All 15 tests passed.

[dch]

above path is not quite correct:

diff --git a/Makefile b/Makefile
index 9758f6e..b195c11 100644
--- a/Makefile
+++ b/Makefile
@@ -1,4 +1,4 @@
-BUILD_CONFIG = build_config.mk
+BUILD_CONFIG = make_config.mk

 all: compile

building with this patch & lz4 -> erocksdb works & has lz4 linked in erocksdb.a building with this patch & without lz4 -> erocksdb works & doesn't have linked in erocksdb.a

\o/