search

facebook / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
28.32k stars 6.29k forks source link

error for compile with -fPIC #7423

Closed software-zjz closed 2 years ago

software-zjz commented 4 years ago

The compilation error is shown below:

root@iZj6c0xle28c53ju72qg5hZ:/usr/local/src/rocksdb# make shared_lib $DEBUG_LEVEL is 0 GEN util/build_version.cc CCLD librocksdb.so.6.13.0 /usr/bin/ld: ./memory/concurrent_arena.o: relocation R_X86_64_TPOFF32 against symbol _ZN7rocksdb15ConcurrentArena9tls_cpuidE' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: ./monitoring/iostats_context.o: relocation R_X86_64_TPOFF32 against symbol_ZN7rocksdb15iostats_contextE' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: ./monitoring/perf_context.o: relocation R_X86_64_TPOFF32 against symbol _ZN7rocksdb12perf_contextE' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: ./monitoring/perf_level.o: relocation R_X86_64_TPOFF32 against symbol_ZN7rocksdb10perf_levelE' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: ./monitoring/thread_status_updater.o: relocation R_X86_64_TPOFF32 against symbol _ZN7rocksdb19ThreadStatusUpdater19thread_status_data_E' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: ./monitoring/thread_status_util.o: relocation R_X86_64_TPOFF32 against symbol_ZN7rocksdb16ThreadStatusUtil27thread_updater_initialized_E' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: ./util/random.o: relocation R_X86_64_TPOFF32 against _ZZN7rocksdb6Random14GetTLSInstanceEvE12tls_instance' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: ./util/thread_local.o: relocation R_X86_64_TPOFF32 against symbol_ZN7rocksdb14ThreadLocalPtr10StaticMeta4tls_E' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: ./db/builder.o: relocation R_X86_64_PC32 against symbol `_ZTVN7rocksdb28CompactionRangeDelAggregatorE' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: error: ld returned 1 exit status Makefile:778: recipe for target 'librocksdb.so.6.13.0' failed make: *** [librocksdb.so.6.13.0] Error 1

Don't know how to solve?

software-zjz commented 4 years ago

cmake version is 3.5.1

mrambacher commented 4 years ago

What OS are you using? Did you do a make clean first?

software-zjz commented 3 years ago

OS is centOs 7,make clean was done, thanks.

brenochoa commented 3 years ago

Getting the same error after upgrading RocksDB version from 6.10.2 to 6.15.4 for compilation inside an Ubuntu-based Docker container.

mrambacher commented 3 years ago

I am having trouble reproducing your problem; I ran a build on Ubuntu without any issues.

Can you see if the -fPIC option is added on your builds (do V=1 make shared_lib). Perhaps delete the object files reported in the message and re-run make with V=1.

Also, are you doing make or cmake?

brenochoa commented 3 years ago

The V=1 doesn't seem to make any difference. Anyway, I can't see the flag being added to the g++ linking step where the process stops. What I have been doing is essentially:

FROM jupyter/minimal-notebook
USER root
RUN apt-get update -y && apt-get -y install build-essential cmake autoconf libsnappy-dev zlib1g-dev libzstd-dev \
            libbz2-dev libgflags-dev liblz4-dev wget liblzma-dev libssl-dev libcurl4-openssl-dev
RUN wget https://github.com/facebook/rocksdb/archive/v6.15.4.tar.gz && tar -xvzf v6.15.4.tar.gz && \
         cd rocksdb-6.15.4 && DEBUG_LEVEL=0 make -j8 shared_lib install-shared

It compiles without issue on 6.10.2, but sometime after 6.12 it breaks. If it helps, the linking step on 6.10.2 does include the -fPIC flag, in fact, without having had to add the V=1, so it looks like some change in the Makefile might have broken it.

yuan-luo commented 3 years ago

I encountered the same issue and my OS is Ubuntu 20.04.

yluo@ubuntu:~/rocksdb$ make shared_lib
$DEBUG_LEVEL is 0
  CCLD     librocksdb.so.6.19.0
/usr/bin/ld: ./memory/concurrent_arena.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb15ConcurrentArena9tls_cpuidE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/iostats_context.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb15iostats_contextE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/perf_context.o: relocation R_X86_64_TPOFF32 against `__tls_guard' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/perf_level.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb10perf_levelE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/thread_status_updater.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb19ThreadStatusUpdater19thread_status_data_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/thread_status_util.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb16ThreadStatusUtil27thread_updater_initialized_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./util/random.o: relocation R_X86_64_TPOFF32 against `_ZZN7rocksdb6Random14GetTLSInstanceEvE12tls_instance' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./util/thread_local.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb14ThreadLocalPtr10StaticMeta4tls_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./cache/cache.o: relocation R_X86_64_PC32 against symbol `_ZN7rocksdb23kDefaultToAdaptiveMutexE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make: *** [Makefile:932: librocksdb.so.6.19.0] Error 1
yluo@ubuntu:~/rocksdb$ uname -a
Linux ubuntu 5.8.0-48-generic #54~20.04.1-Ubuntu SMP Sat Mar 20 13:40:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
yuan-luo commented 3 years ago

Using V=1 got the same error.

yluo@ubuntu:~/rocksdb$ V=1 make shared_lib
$DEBUG_LEVEL is 0
Linux ubuntu 5.8.0-48-generic #54~20.04.1-Ubuntu SMP Sat Mar 20 13:40:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
cc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
g++ -Wl,--no-as-needed -shared -Wl,-soname -Wl,librocksdb.so.6.19  ./cache/cache.o  ./cache/clock_cache.o  ./cache/lru_cache.o  ./cache/sharded_cache.o  ./db/arena_wrapped_db_iter.o  ./db/blob/blob_file_addition.o  ./db/blob/blob_file_builder.o  ./db/blob/blob_file_cache.o  ./db/blob/blob_file_garbage.o  ./db/blob/blob_file_meta.o  ./db/blob/blob_file_reader.o  ./db/blob/blob_log_format.o  ./db/blob/blob_log_sequential_reader.o  ./db/blob/blob_log_writer.o  ./db/builder.o  ./db/c.o  ./db/column_family.o  ./db/compaction/compaction.o  ./db/compaction/compaction_iterator.o  ./db/compaction/compaction_job.o  ./db/compaction/compaction_picker.o  ./db/compaction/compaction_picker_fifo.o  ./db/compaction/compaction_picker_level.o  ./db/compaction/compaction_picker_universal.o  ./db/compaction/sst_partitioner.o  ./db/convenience.o  ./db/db_filesnapshot.o  ./db/db_impl/compacted_db_impl.o  ./db/db_impl/db_impl.o  ./db/db_impl/db_impl_compaction_flush.o  ./db/db_impl/db_impl_debug.o  ./db/db_impl/db_impl_experimental.o  ./db/db_impl/db_impl_files.o  ./db/db_impl/db_impl_open.o  ./db/db_impl/db_impl_readonly.o  ./db/db_impl/db_impl_secondary.o  ./db/db_impl/db_impl_write.o  ./db/db_info_dumper.o  ./db/db_iter.o  ./db/dbformat.o  ./db/error_handler.o  ./db/event_helpers.o  ./db/experimental.o  ./db/external_sst_file_ingestion_job.o  ./db/file_indexer.o  ./db/flush_job.o  ./db/flush_scheduler.o  ./db/forward_iterator.o  ./db/import_column_family_job.o  ./db/internal_stats.o  ./db/logs_with_prep_tracker.o  ./db/log_reader.o  ./db/log_writer.o  ./db/malloc_stats.o  ./db/memtable.o  ./db/memtable_list.o  ./db/merge_helper.o  ./db/merge_operator.o  ./db/output_validator.o  ./db/periodic_work_scheduler.o  ./db/range_del_aggregator.o  ./db/range_tombstone_fragmenter.o  ./db/repair.o  ./db/snapshot_impl.o  ./db/table_cache.o  ./db/table_properties_collector.o  ./db/transaction_log_impl.o  ./db/trim_history_scheduler.o  ./db/version_builder.o  ./db/version_edit.o  ./db/version_edit_handler.o  ./db/version_set.o  ./db/wal_edit.o  ./db/wal_manager.o  ./db/write_batch.o  ./db/write_batch_base.o  ./db/write_controller.o  ./db/write_thread.o  ./env/composite_env.o  ./env/env.o  ./env/env_chroot.o  ./env/env_encryption.o  ./env/env_hdfs.o  ./env/env_posix.o  ./env/file_system.o  ./env/fs_posix.o  ./env/file_system_tracer.o  ./env/io_posix.o  ./env/mock_env.o  ./file/delete_scheduler.o  ./file/file_prefetch_buffer.o  ./file/file_util.o  ./file/filename.o  ./file/line_file_reader.o  ./file/random_access_file_reader.o  ./file/read_write_util.o  ./file/readahead_raf.o  ./file/sequence_file_reader.o  ./file/sst_file_manager_impl.o  ./file/writable_file_writer.o  ./logging/auto_roll_logger.o  ./logging/event_logger.o  ./logging/log_buffer.o  ./memory/arena.o  ./memory/concurrent_arena.o  ./memory/jemalloc_nodump_allocator.o  ./memory/memkind_kmem_allocator.o  ./memtable/alloc_tracker.o  ./memtable/hash_linklist_rep.o  ./memtable/hash_skiplist_rep.o  ./memtable/skiplistrep.o  ./memtable/vectorrep.o  ./memtable/write_buffer_manager.o  ./monitoring/histogram.o  ./monitoring/histogram_windowing.o  ./monitoring/in_memory_stats_history.o  ./monitoring/instrumented_mutex.o  ./monitoring/iostats_context.o  ./monitoring/perf_context.o  ./monitoring/perf_level.o  ./monitoring/persistent_stats_history.o  ./monitoring/statistics.o  ./monitoring/thread_status_impl.o  ./monitoring/thread_status_updater.o  ./monitoring/thread_status_updater_debug.o  ./monitoring/thread_status_util.o  ./monitoring/thread_status_util_debug.o  ./options/cf_options.o  ./options/configurable.o  ./options/customizable.o  ./options/db_options.o  ./options/options.o  ./options/options_helper.o  ./options/options_parser.o  ./port/port_posix.o  ./port/win/env_default.o  ./port/win/env_win.o  ./port/win/io_win.o  ./port/win/port_win.o  ./port/win/win_logger.o  ./port/win/win_thread.o  ./port/stack_trace.o  ./table/adaptive/adaptive_table_factory.o  ./table/block_based/binary_search_index_reader.o  ./table/block_based/block.o  ./table/block_based/block_based_filter_block.o  ./table/block_based/block_based_table_builder.o  ./table/block_based/block_based_table_factory.o  ./table/block_based/block_based_table_iterator.o  ./table/block_based/block_based_table_reader.o  ./table/block_based/block_builder.o  ./table/block_based/block_prefetcher.o  ./table/block_based/block_prefix_index.o  ./table/block_based/data_block_hash_index.o  ./table/block_based/data_block_footer.o  ./table/block_based/filter_block_reader_common.o  ./table/block_based/filter_policy.o  ./table/block_based/flush_block_policy.o  ./table/block_based/full_filter_block.o  ./table/block_based/hash_index_reader.o  ./table/block_based/index_builder.o  ./table/block_based/index_reader_common.o  ./table/block_based/parsed_full_filter_block.o  ./table/block_based/partitioned_filter_block.o  ./table/block_based/partitioned_index_iterator.o  ./table/block_based/partitioned_index_reader.o  ./table/block_based/reader_common.o  ./table/block_based/uncompression_dict_reader.o  ./table/block_fetcher.o  ./table/cuckoo/cuckoo_table_builder.o  ./table/cuckoo/cuckoo_table_factory.o  ./table/cuckoo/cuckoo_table_reader.o  ./table/format.o  ./table/get_context.o  ./table/iterator.o  ./table/merging_iterator.o  ./table/meta_blocks.o  ./table/persistent_cache_helper.o  ./table/plain/plain_table_bloom.o  ./table/plain/plain_table_builder.o  ./table/plain/plain_table_factory.o  ./table/plain/plain_table_index.o  ./table/plain/plain_table_key_coding.o  ./table/plain/plain_table_reader.o  ./table/sst_file_dumper.o  ./table/sst_file_reader.o  ./table/sst_file_writer.o  ./table/table_factory.o  ./table/table_properties.o  ./table/two_level_iterator.o  ./test_util/sync_point.o  ./test_util/sync_point_impl.o  ./test_util/transaction_test_util.o  ./tools/dump/db_dump_tool.o  ./trace_replay/trace_replay.o  ./trace_replay/block_cache_tracer.o  ./trace_replay/io_tracer.o  ./util/build_version.o  ./util/coding.o  ./util/compaction_job_stats_impl.o  ./util/comparator.o  ./util/compression_context_cache.o  ./util/concurrent_task_limiter_impl.o  ./util/crc32c.o  ./util/dynamic_bloom.o  ./util/hash.o  ./util/murmurhash.o  ./util/random.o  ./util/rate_limiter.o  ./util/ribbon_config.o  ./util/slice.o  ./util/file_checksum_helper.o  ./util/status.o  ./util/string_util.o  ./util/thread_local.o  ./util/threadpool_imp.o  ./util/xxhash.o  ./utilities/backupable/backupable_db.o  ./utilities/blob_db/blob_compaction_filter.o  ./utilities/blob_db/blob_db.o  ./utilities/blob_db/blob_db_impl.o  ./utilities/blob_db/blob_db_impl_filesnapshot.o  ./utilities/blob_db/blob_file.o  ./utilities/cassandra/cassandra_compaction_filter.o  ./utilities/cassandra/format.o  ./utilities/cassandra/merge_operator.o  ./utilities/checkpoint/checkpoint_impl.o  ./utilities/compaction_filters/remove_emptyvalue_compactionfilter.o  ./utilities/convenience/info_log_finder.o  ./utilities/debug.o  ./utilities/env_mirror.o  ./utilities/env_timed.o  ./utilities/fault_injection_env.o  ./utilities/fault_injection_fs.o  ./utilities/leveldb_options/leveldb_options.o  ./utilities/memory/memory_util.o  ./utilities/merge_operators/max.o  ./utilities/merge_operators/put.o  ./utilities/merge_operators/sortlist.o  ./utilities/merge_operators/string_append/stringappend.o  ./utilities/merge_operators/string_append/stringappend2.o  ./utilities/merge_operators/uint64add.o  ./utilities/merge_operators/bytesxor.o  ./utilities/object_registry.o  ./utilities/option_change_migration/option_change_migration.o  ./utilities/options/options_util.o  ./utilities/persistent_cache/block_cache_tier.o  ./utilities/persistent_cache/block_cache_tier_file.o  ./utilities/persistent_cache/block_cache_tier_metadata.o  ./utilities/persistent_cache/persistent_cache_tier.o  ./utilities/persistent_cache/volatile_tier_impl.o  ./utilities/simulator_cache/cache_simulator.o  ./utilities/simulator_cache/sim_cache.o  ./utilities/table_properties_collectors/compact_on_deletion_collector.o  ./utilities/trace/file_trace_reader_writer.o  ./utilities/transactions/lock/lock_manager.o  ./utilities/transactions/lock/point/point_lock_tracker.o  ./utilities/transactions/lock/point/point_lock_manager.o  ./utilities/transactions/optimistic_transaction.o  ./utilities/transactions/optimistic_transaction_db_impl.o  ./utilities/transactions/pessimistic_transaction.o  ./utilities/transactions/pessimistic_transaction_db.o  ./utilities/transactions/snapshot_checker.o  ./utilities/transactions/transaction_base.o  ./utilities/transactions/transaction_db_mutex_impl.o  ./utilities/transactions/transaction_util.o  ./utilities/transactions/write_prepared_txn.o  ./utilities/transactions/write_prepared_txn_db.o  ./utilities/transactions/write_unprepared_txn.o  ./utilities/transactions/write_unprepared_txn_db.o  ./utilities/ttl/db_ttl_impl.o  ./utilities/write_batch_with_index/write_batch_with_index.o  ./utilities/write_batch_with_index/write_batch_with_index_internal.o   ./third-party/folly/folly/detail/Futex.o  ./third-party/folly/folly/synchronization/AtomicNotification.o  ./third-party/folly/folly/synchronization/DistributedMutex.o  ./third-party/folly/folly/synchronization/ParkingLot.o  ./third-party/folly/folly/synchronization/WaitOptions.o  ./utilities/transactions/lock/range/range_tree/lib/locktree/concurrent_tree.o  ./utilities/transactions/lock/range/range_tree/lib/locktree/keyrange.o  ./utilities/transactions/lock/range/range_tree/lib/locktree/lock_request.o  ./utilities/transactions/lock/range/range_tree/lib/locktree/locktree.o  ./utilities/transactions/lock/range/range_tree/lib/locktree/manager.o  ./utilities/transactions/lock/range/range_tree/lib/locktree/range_buffer.o  ./utilities/transactions/lock/range/range_tree/lib/locktree/treenode.o  ./utilities/transactions/lock/range/range_tree/lib/locktree/txnid_set.o  ./utilities/transactions/lock/range/range_tree/lib/locktree/wfg.o  ./utilities/transactions/lock/range/range_tree/lib/standalone_port.o  ./utilities/transactions/lock/range/range_tree/lib/util/dbt.o  ./utilities/transactions/lock/range/range_tree/lib/util/memarena.o  ./utilities/transactions/lock/range/range_tree/range_tree_lock_manager.o  ./utilities/transactions/lock/range/range_tree/range_tree_lock_tracker.o  -lpthread -lrt -ldl -lsnappy -lgflags -lz -lbz2 -llz4 -lzstd  -o librocksdb.so.6.19.0
/usr/bin/ld: ./memory/concurrent_arena.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb15ConcurrentArena9tls_cpuidE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/iostats_context.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb15iostats_contextE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/perf_context.o: relocation R_X86_64_TPOFF32 against `__tls_guard' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/perf_level.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb10perf_levelE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/thread_status_updater.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb19ThreadStatusUpdater19thread_status_data_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./monitoring/thread_status_util.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb16ThreadStatusUtil27thread_updater_initialized_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./util/random.o: relocation R_X86_64_TPOFF32 against `_ZZN7rocksdb6Random14GetTLSInstanceEvE12tls_instance' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./util/thread_local.o: relocation R_X86_64_TPOFF32 against symbol `_ZN7rocksdb14ThreadLocalPtr10StaticMeta4tls_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ./cache/cache.o: relocation R_X86_64_PC32 against symbol `_ZN7rocksdb23kDefaultToAdaptiveMutexE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make: *** [Makefile:932: librocksdb.so.6.19.0] Error 1
caunt commented 3 years ago

Try this way.


make shared_lib
make static_lib
make install-shared```
pdillinger commented 3 years ago

You have to make clean if you change between shared or static library builds (as Mark said above, and some claim to have done)

mrambacher commented 3 years ago

@bernochoa I think I understand the problem. I believe it is a problem only if you have multiple make goals. What happens if you execute the command in two steps: DEBUG_LEVEL=0 make -j8 shared_lib DEBUG_LEVEL=0 make install-shared Does it work in that case?