search

duckdb / sqlite_scanner

DuckDB extension to read and write to SQLite databases
MIT License
189 stars 19 forks source link

Can't build on an M1/ARM Mac #80

Open cpaika opened 5 months ago

cpaika commented 5 months ago

What happens?

I'm attempting to clone this repo to make changes. I cloned the repo, and initialized the duckdb submodule. When I run make it fails 82% of the way in building the parquet duckdb extension:

[ 82%] Built target tpch_loadable_extension
[ 82%] Linking CXX shared library parquet.duckdb_extension
Undefined symbols for architecture arm64:
  "_parquet_init", referenced from:
     -exported_symbol[s_list] command line option
  "_parquet_version", referenced from:
     -exported_symbol[s_list] command line option
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [extension/parquet/parquet.duckdb_extension] Error 1
make[2]: *** [extension/parquet/CMakeFiles/parquet_loadable_extension.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [release] Error 2

The same error occurs when I run GEN=ninja make as well.

To Reproduce

Clone this repository, initialize the submodule, and run make

OS:

Mac OS X

SQLite Version:

Default

DuckDB Version:

main

DuckDB Client:

N/A

Full Name:

Christopher Paika

Affiliation:

Tesla

Have you tried this on the latest main branch?

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

Mytherin commented 5 months ago

Thanks for the report! I'm running on an M1 mac as well and not running into this problem - are you perhaps on an older MacOS/compiler version or using a custom/different compiler?

$> clang++ --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Alternatively do you perhaps have any compilation flags defined in your env that could change symbol visibility/linking behavior?

cpaika commented 5 months ago

Upgraded to Sonoma to match your clang++ version, still getting the exception unfortunately:

Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

My InstalledDir is different, unsure if that matters if the versions are the same.

It seems like the problem is building the duckdb submodule, is there anything special I have to do to that other than check it out?

Mytherin commented 5 months ago

Hm, odd. I still suspect there's something different about the configuration/environment setup that causes the change in behavior. As a work-around you could try uncommenting this line which is likely where the problem originates from.

breckcs commented 5 months ago

I work for the same organization as @cpaika, so I have a similar M1 Mac setup. I can build duckdb (build/release/duckdb) and the sqlite_scanner extension (build/release/extension/sqlite_scanner/sqlite_scanner.duckdb_extension). My build fails later linking the parquet.duckdb_extension:

[ 91%] Linking CXX shared library parquet.duckdb_extension
ld: warning: ignoring duplicate libraries: '../../src/libduckdb_static.a', '../tpch/libtpch_extension.a', 'libparquet_extension.a'
ld: Undefined symbols:
  _parquet_init, referenced from:
      <initial-undefines>
  _parquet_version, referenced from:
      <initial-undefines>
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[3]: *** [extension/parquet/CMakeFiles/parquet_loadable_extension.dir/build.make:721: extension/parquet/parquet.duckdb_extension] Error 1
gmake[2]: *** [CMakeFiles/Makefile2:7770: extension/parquet/CMakeFiles/parquet_loadable_extension.dir/all] Error 2
gmake[1]: *** [Makefile:136: all] Error 2
gmake[1]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
make: *** [release] Error 2

The steps I used:

  1. git clone https://github.com/duckdb/sqlite_scanner.git
  2. cd sqlite_scanner
  3. git submodule init
  4. git submodule update
  5. make

My clang++ version:

% clang++ --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

@cpaika, the difference in InstalledDir may be because you don't have Xcode installed.

carlopi commented 5 months ago

I think this might come from the fact that somehow the linker links against a [lib]parquet_extension.so or some similar problem connected to dynamic libraries vs static libraries.

Can you try check something what the command ls build/relese/**/*.so returns? Or add some more logging to the build?

breckcs commented 5 months ago

If I comment out this line, it builds.

That line was originally introduced by @jkub here, then modified by @Mytherin here then here.

With CMAKE_VERBOSE_MAKEFILE set to ON, the output when the build works:

[ 91%] Linking CXX shared library parquet.duckdb_extension
cd /Users/cbreck/github/sqlite_scanner/build/release/extension/parquet && /opt/homebrew/Cellar/cmake/3.28.1/bin/cmake -E cmake_link_script CMakeFiles/parquet_loadable_extension.dir/link.txt --verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -O3 -DNDEBUG   -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk -dynamiclib -Wl,-headerpad_max_install_names -o parquet.duckdb_extension -install_name @rpath/parquet.duckdb_extension CMakeFiles/parquet_loadable_extension.dir/column_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/column_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_crypto.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_extension.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_metadata.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_statistics.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_timestamp.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/serialize_parquet.cpp.o CMakeFiles/parquet_loadable_extension.dir/zstd_file_system.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_constants.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_types.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/protocol/TProtocol.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TTransportException.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TBufferTransports.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy.cc.o "CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy-sinksource.cc.o" CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_ddict.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/huf_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress_block.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/entropy_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/fse_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/zstd_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/error_private.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/xxhash.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/fse_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/hist.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/huf_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_literals.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_sequences.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_superblock.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_double_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_lazy.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_ldm.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_opt.cpp.o  ../../src/libduckdb_static.a -Wl,-dead_strip ../tpch/libtpch_extension.a libparquet_extension.a ../../src/libduckdb_static.a ../tpch/libtpch_extension.a libparquet_extension.a ../../third_party/fsst/libduckdb_fsst.a ../../third_party/fmt/libduckdb_fmt.a ../../third_party/libpg_query/libduckdb_pg_query.a ../../third_party/re2/libduckdb_re2.a ../../third_party/miniz/libduckdb_miniz.a ../../third_party/utf8proc/libduckdb_utf8proc.a ../../third_party/hyperloglog/libduckdb_hyperloglog.a ../../third_party/fastpforlib/libduckdb_fastpforlib.a ../../third_party/skiplist/libduckdb_skiplistlib.a ../../third_party/mbedtls/libduckdb_mbedtls.a
ld: warning: ignoring duplicate libraries: '../../src/libduckdb_static.a', '../tpch/libtpch_extension.a', 'libparquet_extension.a'
gmake[3]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'

And when it fails:

[ 91%] Linking CXX shared library parquet.duckdb_extension
cd /Users/cbreck/github/sqlite_scanner/build/release/extension/parquet && /opt/homebrew/Cellar/cmake/3.28.1/bin/cmake -E cmake_link_script CMakeFiles/parquet_loadable_extension.dir/link.txt --verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -O3 -DNDEBUG   -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk -dynamiclib -Wl,-headerpad_max_install_names -o parquet.duckdb_extension -install_name @rpath/parquet.duckdb_extension CMakeFiles/parquet_loadable_extension.dir/column_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/column_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_crypto.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_extension.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_metadata.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_statistics.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_timestamp.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/serialize_parquet.cpp.o CMakeFiles/parquet_loadable_extension.dir/zstd_file_system.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_constants.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_types.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/protocol/TProtocol.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TTransportException.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TBufferTransports.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy.cc.o "CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy-sinksource.cc.o" CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_ddict.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/huf_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress_block.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/entropy_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/fse_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/zstd_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/error_private.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/xxhash.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/fse_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/hist.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/huf_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_literals.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_sequences.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_superblock.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_double_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_lazy.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_ldm.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_opt.cpp.o  ../../src/libduckdb_static.a -Wl,-dead_strip -Wl,-exported_symbol,_parquet_init -Wl,-exported_symbol,_parquet_version -Wl,-exported_symbol,_parquet_storage_ini* ../tpch/libtpch_extension.a libparquet_extension.a ../../src/libduckdb_static.a ../tpch/libtpch_extension.a libparquet_extension.a ../../third_party/fsst/libduckdb_fsst.a ../../third_party/fmt/libduckdb_fmt.a ../../third_party/libpg_query/libduckdb_pg_query.a ../../third_party/re2/libduckdb_re2.a ../../third_party/miniz/libduckdb_miniz.a ../../third_party/utf8proc/libduckdb_utf8proc.a ../../third_party/hyperloglog/libduckdb_hyperloglog.a ../../third_party/fastpforlib/libduckdb_fastpforlib.a ../../third_party/skiplist/libduckdb_skiplistlib.a ../../third_party/mbedtls/libduckdb_mbedtls.a
ld: warning: ignoring duplicate libraries: '../../src/libduckdb_static.a', '../tpch/libtpch_extension.a', 'libparquet_extension.a'
ld: Undefined symbols:
  _parquet_init, referenced from:
      <initial-undefines>
  _parquet_version, referenced from:
      <initial-undefines>
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[3]: *** [extension/parquet/CMakeFiles/parquet_loadable_extension.dir/build.make:724: extension/parquet/parquet.duckdb_extension] Error 1
gmake[3]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
gmake[2]: *** [CMakeFiles/Makefile2:7773: extension/parquet/CMakeFiles/parquet_loadable_extension.dir/all] Error 2
gmake[2]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
gmake[1]: *** [Makefile:139: all] Error 2
gmake[1]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
make: *** [release] Error 2