[C++] Errors from jemalloc when building pyarrow from source on OSX and Debian

[asfimport]

My attempts to build pyarrow from source are failing. I've set up the conda environment using the instructions provided in the Develop instructions, and have tried this on both Debian and OSX. When I run CMAKE in debug mode on OSX, the output is:

-- Building using CMake version: 3.14.0
-- Arrow version: 0.13.0 (full: '0.13.0-SNAPSHOT')
-- clang-tidy not found
-- clang-format not found
-- infer found at /usr/local/bin/infer
-- Using ccache: /usr/local/bin/ccache
-- Found cpplint executable at /Users/Greg/documents/repos/arrow/cpp/build-support/cpplint.py
-- Compiler command: env LANG=C /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -v
-- Compiler version: Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

-- Compiler id: AppleClang
Selected compiler clang 4.1.0svn
-- Arrow build warning level: CHECKIN
Configured for DEBUG build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: DEBUG
-- BOOST_VERSION: 1.67.0
-- BROTLI_VERSION: v0.6.0
-- CARES_VERSION: 1.15.0
-- DOUBLE_CONVERSION_VERSION: v3.1.1
-- FLATBUFFERS_VERSION: v1.10.0
-- GBENCHMARK_VERSION: v1.4.1
-- GFLAGS_VERSION: v2.2.0
-- GLOG_VERSION: v0.3.5
-- GRPC_VERSION: v1.18.0
-- GTEST_VERSION: 1.8.1
-- JEMALLOC_VERSION: 17c897976c60b0e6e4f4a365c751027244dada7a
-- LZ4_VERSION: v1.8.3
-- ORC_VERSION: 1.5.4
-- PROTOBUF_VERSION: v3.6.1
-- RAPIDJSON_VERSION: v1.1.0
-- RE2_VERSION: 2018-10-01
-- SNAPPY_VERSION: 1.1.3
-- THRIFT_VERSION: 0.11.0
-- ZLIB_VERSION: 1.2.8
-- ZSTD_VERSION: v1.3.7
-- Boost version: 1.68.0
-- Found the following Boost libraries:
--   regex
--   system
--   filesystem
-- Boost include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
-- Boost libraries: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_regex.dylib/Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_system.dylib/Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_filesystem.dylib
Added shared library dependency boost_system_shared: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_system.dylib
Added shared library dependency boost_filesystem_shared: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_filesystem.dylib
Added shared library dependency boost_regex_shared: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_regex.dylib
Added static library dependency double-conversion_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libdouble-conversion.a
-- double-conversion include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
-- double-conversion static library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libdouble-conversion.a
-- GFLAGS_HOME: /Users/Greg/anaconda3/envs/pyarrow-dev
-- GFlags include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
-- GFlags static library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libgflags.a
Added static library dependency gflags_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libgflags.a
-- RapidJSON include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
-- Found the Flatbuffers library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libflatbuffers.a
-- Flatbuffers include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
-- Flatbuffers compiler: /Users/Greg/anaconda3/envs/pyarrow-dev/bin/flatc
Added static library dependency jemalloc_static: /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist//lib/libjemalloc_pic.a
Added shared library dependency jemalloc_shared: /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist//lib/libjemalloc.dylib
-- Found hdfs.h at: /Users/Greg/documents/repos/arrow/cpp/thirdparty/hadoop/include/hdfs.h
-- Found the ZLIB shared library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libz.dylib
Added shared library dependency zlib_shared: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libz.dylib
-- SNAPPY_HOME: /Users/Greg/anaconda3/envs/pyarrow-dev
-- Found the Snappy library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libsnappy.dylib
Added static library dependency snappy_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libsnappy.a
-- Found the Brotli library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlienc-static.a;/Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlidec-static.a;/Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlicommon-static.a
Added static library dependency brotli_enc_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlienc-static.a
Added static library dependency brotli_dec_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlidec-static.a
Added static library dependency brotli_common_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlicommon-static.a
Added static library dependency lz4_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/liblz4.a
Added static library dependency zstd_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libzstd.a
-- THRIFT_HOME: /Users/Greg/anaconda3/envs/pyarrow-dev
-- Thrift version: 0.12.0
-- Thrift include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
-- Thrift static library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libthrift.a
-- Thrift compiler: /Users/Greg/anaconda3/envs/pyarrow-dev/bin/thrift
-- GLOG_HOME: /Users/Greg/anaconda3/envs/pyarrow-dev
-- Found the GLog library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libglog.dylib
-- Glog include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
-- Glog static library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libglog.a
Added static library dependency glog_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libglog.a
-- CMAKE_C_FLAGS:  -Qunused-arguments -ggdb -O0   -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-deprecated -Wno-weak-vtables -Wno-padded -Wno-comma -Wno-unused-macros -Wno-unused-parameter -Wno-unused-template -Wno-undef -Wno-shadow -Wno-switch-enum -Wno-exit-time-destructors -Wno-global-constructors -Wno-weak-template-vtables -Wno-undefined-reinterpret-cast -Wno-implicit-fallthrough -Wno-unreachable-code-return -Wno-float-equal -Wno-missing-prototypes -Wno-documentation-unknown-command -Wno-old-style-cast -Wno-covered-switch-default -Wno-cast-align -Wno-vla-extension -Wno-shift-sign-overflow -Wno-used-but-marked-unused -Wno-missing-variable-declarations -Wno-gnu-zero-variadic-macro-arguments -Wconversion -Wno-sign-conversion -Wno-disabled-macro-expansion -Wno-format-nonliteral -Wno-missing-noreturn -Wno-gnu-folding-constant -Wno-reserved-id-macro -Wno-range-loop-analysis -Wno-double-promotion -Wno-undefined-func-template -Wno-zero-as-null-pointer-constant -Wno-unknown-warning-option -Werror -Wno-unknown-warning-option -msse4.2 -maltivec -stdlib=libc++
-- CMAKE_CXX_FLAGS:   -Qunused-arguments -ggdb -O0  -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-deprecated -Wno-weak-vtables -Wno-padded -Wno-comma -Wno-unused-macros -Wno-unused-parameter -Wno-unused-template -Wno-undef -Wno-shadow -Wno-switch-enum -Wno-exit-time-destructors -Wno-global-constructors -Wno-weak-template-vtables -Wno-undefined-reinterpret-cast -Wno-implicit-fallthrough -Wno-unreachable-code-return -Wno-float-equal -Wno-missing-prototypes -Wno-documentation-unknown-command -Wno-old-style-cast -Wno-covered-switch-default -Wno-cast-align -Wno-vla-extension -Wno-shift-sign-overflow -Wno-used-but-marked-unused -Wno-missing-variable-declarations -Wno-gnu-zero-variadic-macro-arguments -Wconversion -Wno-sign-conversion -Wno-disabled-macro-expansion -Wno-format-nonliteral -Wno-missing-noreturn -Wno-gnu-folding-constant -Wno-reserved-id-macro -Wno-range-loop-analysis -Wno-double-promotion -Wno-undefined-func-template -Wno-zero-as-null-pointer-constant -Wno-unknown-warning-option -Werror -Wno-unknown-warning-option -msse4.2 -maltivec -stdlib=libc++ 

-- Configuring done
CMake Warning (dev):
  Policy CMP0068 is not set: RPATH settings on macOS do not affect
  install_name.  Run "cmake --help-policy CMP0068" for policy details.  Use
  the cmake_policy command to set the policy and suppress this warning.

  For compatibility with older versions of CMake, the install_name fields for
  the following targets are still affected by RPATH settings:

   arrow_python_shared
   arrow_shared
   parquet_shared
   plasma_shared

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/Greg/documents/repos/arrow/cpp/build

After I run make -j4, the output is:

  0%] Built target metadata_fbs
[  1%] Performing configure step for 'jemalloc_ep'
[  2%] Built target parquet-thrift-deps
[ 12%] Built target arrow_python_objlib
[ 19%] Built target plasma_objlib
[ 19%] Built target arrow_python_static
CMake Error at /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-configure-DEBUG.cmake:49 (message):
  Command failed: 1

   './autogen.sh' 'AR=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar' 'CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc' '--prefix=/Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist/' '--with-jemalloc-prefix=je_arrow_' '--with-private-namespace=je_arrow_private_' '--disable-tls'

  See also

    /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-configure-*.log

make[2]: *** [jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-configure] Error 1
make[1]: *** [CMakeFiles/jemalloc_ep.dir/all] Error 2
make: *** [all] Error 2

And the output of the jemalloc log file is:

configure: error: in `/Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
autoconf
./configure --enable-autogen AR=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc --prefix=/Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist/ --with-jemalloc-prefix=je_arrow_ --with-private-namespace=je_arrow_private_ --disable-tls
checking for xsltproc... /usr/bin/xsltproc
checking for gcc... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... Error 0 in ./configure

Any help / guidance I can get is appreciated.

Environment: OSX, Debian, Python==3.6.7 Reporter: Gregory Hayes

Externally tracked issue: https://github.com/apache/arrow/issues/3948

_{Note: This issue was originally created as ARROW-4935. Please see the migration documentation for further details.}

[asfimport]

Uwe Korn / @xhochy: [~hayesgb] It seems that you are no on latest master. I would recommend pulling the sources again as we have made a lot of changes to the build system in the last days. They might already help.

Otherwise, you seem to be inside a conda environment with the latest build settings. Here you should nowadays use the compilers that are provided by Anaconda. You can install them using the compilers package from conda-forge: conda install -c conda-forge compilers and then reactivate your conda environment. On OSX, this also requires that you have the 10.9 OSX SDK downloaded, e.g. to /opt/MacOSX10.9.sdk and then set export CONDA_BUILD_SYSROOT=/opt/MacOSX10.9.sdk before running a new build (delete your build directory to be safe that all new settings are picked up).

Can you upload also the config.log produced by jemalloc if it fails again?

[asfimport]

Gregory Hayes: Not sure how I ended up at that documentation, but thank you for pointing me in the right direction.  

I created download from git, and re-created my conda environment.  Initially, I got an error that LLVM v7 was not available, and manually installed LLVM-META using conda, which eliminated that error.  I'm still getting jemalloc errors though.  Log file shown below:

install: mkdir /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist: File exists
make[3]: *** [install_include] Error 71
make[3]: *** Waiting for unfinished jobs....
/usr/bin/install -c -m 755 bin/jemalloc-config /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist/bin
/usr/bin/install -c -m 755 bin/jemalloc.sh /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist/bin
/usr/bin/install -c -m 755 bin/jeprof /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist/bin

[asfimport]

Uwe Korn / @xhochy: This seems like a race condition in the jemalloc build. Does it work when you delete  /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist and then build again?

[asfimport]

Gregory Hayes: That worked to get me past the jemalloc error. Now I'm seeing an unable to locate gandiva library error, when I try to run setup.py. This happens even if I flag GANDIVA =OFF and set

export PYARROW_WITH_GANDIVA=0

[asfimport]

Uwe Korn / @xhochy: Please always post the error, otherwise we are unable to see what the problem could be.

[asfimport]

Gregory Hayes: The immediate error is:

-- Configuring incomplete, errors occurred!
See also "/Users/Greg/Documents/repos/arrow/python/build/temp.macosx-10.9-x86_64-3.7/CMakeFiles/CMakeOutput.log".
See also "/Users/Greg/Documents/repos/arrow/python/build/temp.macosx-10.9-x86_64-3.7/CMakeFiles/CMakeError.log".
error: command 'cmake' failed with exit status 1

Output of the CMakeError.log file is:

Performing C++ SOURCE FILE Test CXX_SUPPORTS_ALTIVEC failed with the following output:
Change Dir: /Users/Greg/Documents/repos/arrow/python/build/temp.macosx-10.9-x86_64-3.7/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_706c2/fast 
/Library/Developer/CommandLineTools/usr/bin/make -f CMakeFiles/cmTC_706c2.dir/build.make CMakeFiles/cmTC_706c2.dir/build
Building CXX object CMakeFiles/cmTC_706c2.dir/src.cxx.o
/Users/Greg/anaconda3/envs/pyarrow-dev/bin/x86_64-apple-darwin13.4.0-clang++    -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -std=c++14 -fmessage-length=0 -DCXX_SUPPORTS_ALTIVEC -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.9   -maltivec -o CMakeFiles/cmTC_706c2.dir/src.cxx.o -c /Users/Greg/Documents/repos/arrow/python/build/temp.macosx-10.9-x86_64-3.7/CMakeFiles/CMakeTmp/src.cxx
clang-4.0: error: invalid argument '-faltivec' only allowed with 'ppc/ppc64/ppc64le'
make[1]: *** [CMakeFiles/cmTC_706c2.dir/src.cxx.o] Error 1
make: *** [cmTC_706c2/fast] Error 2

Source file was:
int main() { return 0; }
Performing C++ SOURCE FILE Test CXX_SUPPORTS_ARMCRC failed with the following output:
Change Dir: /Users/Greg/Documents/repos/arrow/python/build/temp.macosx-10.9-x86_64-3.7/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_c6418/fast 
/Library/Developer/CommandLineTools/usr/bin/make -f CMakeFiles/cmTC_c6418.dir/build.make CMakeFiles/cmTC_c6418.dir/build
Building CXX object CMakeFiles/cmTC_c6418.dir/src.cxx.o
/Users/Greg/anaconda3/envs/pyarrow-dev/bin/x86_64-apple-darwin13.4.0-clang++    -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -std=c++14 -fmessage-length=0 -DCXX_SUPPORTS_ARMCRC -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.9   -march=armv8-a+crc -o CMakeFiles/cmTC_c6418.dir/src.cxx.o -c /Users/Greg/Documents/repos/arrow/python/build/temp.macosx-10.9-x86_64-3.7/CMakeFiles/CMakeTmp/src.cxx
error: unknown target CPU 'armv8-a+crc'
make[1]: *** [CMakeFiles/cmTC_c6418.dir/src.cxx.o] Error 1
make: *** [cmTC_c6418/fast] Error 2

Source file was:
int main() { return 0; }

[asfimport]

Uwe Korn / @xhochy: [~hayesgb] The CMakeError.log is fine, these are simply error where we test your system for features that it doesn't have. No need to worry about this. Configuring incomplete, errors occurred! indicates that CMake had errors and should have displayed it somewhere in the command line output (CMake doesn't stop at errors).

[asfimport]

Gregory Hayes: I was able to successfully build the and run setup.py. Interestingly enough, after getting cmake to run successfully, I ran into problems with setup.py. The error reported was:

 Run Build Command(s):/usr/bin/make cmTC_deb6c/fast 
    /Library/Developer/CommandLineTools/usr/bin/make -f CMakeFiles/cmTC_deb6c.dir/build.make CMakeFiles/cmTC_deb6c.dir/build
    Building C object CMakeFiles/cmTC_deb6c.dir/testCCompiler.c.o
    /Users/Greg/anaconda3/envs/pyarrow-dev/bin/x86_64-apple-darwin13.4.0-clang   -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.9   -o CMakeFiles/cmTC_deb6c.dir/testCCompiler.c.o   -c /Users/Greg/Documents/repos/arrow/python/build/temp.macosx-10.9-x86_64-3.7/CMakeFiles/CMakeTmp/testCCompiler.c
    Linking C executable cmTC_deb6c
    /Users/Greg/anaconda3/envs/pyarrow-dev/bin/cmake -E cmake_link_script CMakeFiles/cmTC_deb6c.dir/link.txt --verbose=1
    /Users/Greg/anaconda3/envs/pyarrow-dev/bin/x86_64-apple-darwin13.4.0-clang -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.9 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs  CMakeFiles/cmTC_deb6c.dir/testCCompiler.c.o  -o cmTC_deb6c 
    ld: warning: ignoring file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/lib/libSystem.tbd, file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/lib/libSystem.tbd
    ld: dynamic main executables must link with libSystem.dylib for architecture x86_64
    clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
    make[1]: *** [cmTC_deb6c] Error 1
    make: *** [cmTC_deb6c/fast] Error 2

Following this link suggests the issue is $CONDA_BUILD_SYSROOT needs to be set to the MacOS10.9 sdk, and that it should be set in conda_build_config.yaml, but I could not find that file in the installation directory. However, if I set it manually, then the setup.py file runs fine.

[asfimport]

Uwe Korn / @xhochy: The conda_build_config.yaml only works when you build with conda build but in this case, we aren't using it but calling cmake / the compiler directly, thus you need to export CONDA_BUILD_SYSROOT=<sdk-location>.

[asfimport]

Ian Mateus Vieira Manor: Having the same problem running

cmake install

on OSX, but have no

/build/jemalloc_ep-prefix/src/jemalloc_ep/dist

 directory to delete.

[asfimport]

Ian Mateus Vieira Manor: Solved the jemalloc problem on my machine by installing macOS SDK headers.

cd /Library/Developer/CommandLineTools/Packages/
open macOS_SDK_headers_for_macOS_10.14.pkg

[asfimport]

Todd Farmer / @toddfarmer: Transitioning issue from Resolved to Closed to based on resolution field value.