search

cylondata / cylon

Cylon is a fast, scalable, distributed memory, parallel runtime with a Pandas like DataFrame.
https://cylondata.org
Apache License 2.0
297 stars 48 forks source link

build: Build failure debug mode in MacOS #625

Open nirandaperera opened 1 year ago

nirandaperera commented 1 year ago

Originally reported by @mstaylor 

====================[ Build | cylon | Debug ]===================================
/Applications/CLion.app/Contents/bin/cmake/mac/bin/cmake --build /Users/mstaylor/Documents/phd/code/cylon/cpp/cmake-build-debug --target cylon -j 8
[1/2] Linking CXX shared library lib/libcylon.0.5.0.dylib
FAILED: lib/libcylon.0.5.0.dylib
: && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -Wall -Wextra -Wno-error=redundant-move  -msse4.2 -g -g -fsanitize=address -fno-omit-frame-pointer -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -dynamiclib -Wl,-headerpad_max_install_names -Wl   -current_version 0.5.0 -o lib/libcylon.0.5.0.dylib -install_name @rpath/libcylon.0.5.0.dylib src/cylon/CMakeFiles/cylon.dir/arrow/arrow_all_to_all.cpp.o src/cylon/CMakeFiles/cylon.dir/arrow/arrow_buffer.cpp.o src/cylon/CMakeFiles/cylon.dir/arrow/arrow_builder.cpp.o src/cylon/CMakeFiles/cylon.dir/arrow/arrow_comparator.cpp.o src/cylon/CMakeFiles/cylon.dir/arrow/arrow_kernels.cpp.o src/cylon/CMakeFiles/cylon.dir/arrow/arrow_partition_kernels.cpp.o src/cylon/CMakeFiles/cylon.dir/arrow/arrow_task_all_to_all.cpp.o src/cylon/CMakeFiles/cylon.dir/arrow/arrow_types.cpp.o src/cylon/CMakeFiles/cylon.dir/column.cpp.o src/cylon/CMakeFiles/cylon.dir/compute/aggregate_kernels.cpp.o src/cylon/CMakeFiles/cylon.dir/compute/aggregates.cpp.o src/cylon/CMakeFiles/cylon.dir/compute/scalar_aggregate.cpp.o src/cylon/CMakeFiles/cylon.dir/ctx/arrow_memory_pool_utils.cpp.o src/cylon/CMakeFiles/cylon.dir/ctx/cylon_context.cpp.o src/cylon/CMakeFiles/cylon.dir/data_types.cpp.o src/cylon/CMakeFiles/cylon.dir/groupby/groupby.cpp.o src/cylon/CMakeFiles/cylon.dir/groupby/hash_groupby.cpp.o src/cylon/CMakeFiles/cylon.dir/groupby/pipeline_groupby.cpp.o src/cylon/CMakeFiles/cylon.dir/indexing/index.cpp.o src/cylon/CMakeFiles/cylon.dir/indexing/index_utils.cpp.o src/cylon/CMakeFiles/cylon.dir/indexing/indexer.cpp.o src/cylon/CMakeFiles/cylon.dir/indexing/slice.cpp.o src/cylon/CMakeFiles/cylon.dir/io/arrow_io.cpp.o src/cylon/CMakeFiles/cylon.dir/io/csv_read_config.cpp.o src/cylon/CMakeFiles/cylon.dir/io/csv_write_config.cpp.o src/cylon/CMakeFiles/cylon.dir/io/parquet_config.cpp.o src/cylon/CMakeFiles/cylon.dir/join/hash_join.cpp.o src/cylon/CMakeFiles/cylon.dir/join/join.cpp.o src/cylon/CMakeFiles/cylon.dir/join/join_utils.cpp.o src/cylon/CMakeFiles/cylon.dir/join/sort_join.cpp.o src/cylon/CMakeFiles/cylon.dir/mapreduce/mapreduce.cpp.o src/cylon/CMakeFiles/cylon.dir/net/communicator.cpp.o src/cylon/CMakeFiles/cylon.dir/net/mpi/mpi_channel.cpp.o src/cylon/CMakeFiles/cylon.dir/net/mpi/mpi_communicator.cpp.o src/cylon/CMakeFiles/cylon.dir/net/mpi/mpi_operations.cpp.o src/cylon/CMakeFiles/cylon.dir/net/ops/all_to_all.cpp.o src/cylon/CMakeFiles/cylon.dir/net/ops/gather.cpp.o src/cylon/CMakeFiles/cylon.dir/net/ops/bcast.cpp.o src/cylon/CMakeFiles/cylon.dir/net/cylon_request.cpp.o src/cylon/CMakeFiles/cylon.dir/ops.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/all_to_all_op.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/api/parallel_op.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/dis_join_op.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/dis_set_op.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/execution/execution.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/join_op.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/kernels/join_kernel.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/kernels/partition.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/kernels/prepare_array.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/kernels/row_comparator.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/kernels/set_kernel.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/merge_op.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/partition_op.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/set_op.cpp.o src/cylon/CMakeFiles/cylon.dir/ops/split_op.cpp.o src/cylon/CMakeFiles/cylon.dir/partition/partition.cpp.o src/cylon/CMakeFiles/cylon.dir/row.cpp.o src/cylon/CMakeFiles/cylon.dir/serialize/table_serialize.cpp.o src/cylon/CMakeFiles/cylon.dir/table.cpp.o src/cylon/CMakeFiles/cylon.dir/table_api.cpp.o src/cylon/CMakeFiles/cylon.dir/util/arrow_utils.cpp.o src/cylon/CMakeFiles/cylon.dir/util/builtins.cpp.o src/cylon/CMakeFiles/cylon.dir/util/copy_arrray.cpp.o src/cylon/CMakeFiles/cylon.dir/util/flatten_array.cpp.o src/cylon/CMakeFiles/cylon.dir/util/logging.cpp.o src/cylon/CMakeFiles/cylon.dir/util/murmur3.cpp.o src/cylon/CMakeFiles/cylon.dir/util/uuid.cpp.o src/cylon/CMakeFiles/cylon.dir/scalar.cpp.o src/cylon/CMakeFiles/cylon.dir/net/ops/base_ops.cpp.o src/cylon/CMakeFiles/cylon.dir/net/utils.cpp.o -L/Users/mstaylor/opt/anaconda3/envs/cylon_dev/lib   -L/Users/mstaylor/opt/anaconda3/envs/cylon_dev/Library/lib -Wl,-rpath,/Users/mstaylor/opt/anaconda3/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/Users/mstaylor/opt/anaconda3/envs/cylon_dev/lib -Wl,-rpath,/Users/mstaylor/opt/anaconda3/envs/cylon_dev/Library/lib  /Users/mstaylor/opt/anaconda3/lib/libmpicxx.dylib  /Users/mstaylor/opt/anaconda3/lib/libmpi.dylib  /Users/mstaylor/opt/anaconda3/lib/libpmpi.dylib  /usr/local/lib/libglog.0.6.0.dylib  /Users/mstaylor/opt/anaconda3/envs/cylon_dev/lib/libarrow.dylib  /Users/mstaylor/opt/anaconda3/envs/cylon_dev/lib/libparquet.dylib  /usr/local/lib/libgflags.2.2.2.dylib && :
Undefined symbols for architecture x86_64:
  “_ompi_mpi_byte”, referenced from:
      cylon::MPIChannel::progressReceives() in mpi_channel.cpp.o
      cylon::MPIChannel::progressSends() in mpi_channel.cpp.o
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
  “_ompi_mpi_c_bool”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
  “_ompi_mpi_comm_null”, referenced from:
      cylon::net::MPICommunicator::Make(std::__1::shared_ptr<cylon::net::CommConfig> const&, cylon::MemoryPool*, std::__1::shared_ptr<cylon::net::Communicator>*) in mpi_communicator.cpp.o
  “_ompi_mpi_comm_world”, referenced from:
      cylon::net::MPICommunicator::Make(std::__1::shared_ptr<cylon::net::CommConfig> const&, cylon::MemoryPool*, std::__1::shared_ptr<cylon::net::Communicator>*) in mpi_communicator.cpp.o
  “_ompi_mpi_datatype_null”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
      cylon::mpi::AllReduce(std::__1::shared_ptr<cylon::CylonContext> const&, void const*, void*, int, std::__1::shared_ptr<cylon::DataType> const&, cylon::net::ReduceOp) in mpi_operations.cpp.o
      cylon::mpi::MpiAllReduceImpl::AllReduceBuffer(void const*, void*, int, std::__1::shared_ptr<cylon::DataType> const&, cylon::net::ReduceOp) const in mpi_operations.cpp.o
  “_ompi_mpi_double”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
  “_ompi_mpi_errors_return”, referenced from:
      cylon::net::MPICommunicator::Make(std::__1::shared_ptr<cylon::net::CommConfig> const&, cylon::MemoryPool*, std::__1::shared_ptr<cylon::net::Communicator>*) in mpi_communicator.cpp.o
  “_ompi_mpi_float”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
  “_ompi_mpi_int”, referenced from:
      cylon::MPIChannel::init(int, std::__1::vector<int, std::__1::allocator<int> > const&, std::__1::vector<int, std::__1::allocator<int> > const&, cylon::ChannelReceiveCallback*, cylon::ChannelSendCallback*, cylon::Allocator*) in mpi_channel.cpp.o
      cylon::MPIChannel::progressReceives() in mpi_channel.cpp.o
      cylon::MPIChannel::sendHeader(std::__1::pair<int const, cylon::PendingSend*> const&) const in mpi_channel.cpp.o
      cylon::MPIChannel::sendFinishHeader(std::__1::pair<int const, cylon::PendingSend*> const&) const in mpi_channel.cpp.o
  “_ompi_mpi_int16_t”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
  “_ompi_mpi_int32_t”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
      cylon::mpi::MpiTableGatherImpl::GatherBufferSizes(int const*, int, int*, int) const in gather.cpp.o
      cylon::mpi::GatherArrowBuffer(std::__1::shared_ptr<arrow::Buffer> const&, int, std::__1::shared_ptr<cylon::CylonContext> const&, std::__1::vector<std::__1::shared_ptr<arrow::Buffer>, std::__1::allocator<std::__1::shared_ptr<arrow::Buffer> > >&) in gather.cpp.o
      cylon::mpi::MpiTableAllgatherImpl::AllgatherBufferSizes(int const*, int, int*) const in gather.cpp.o
      cylon::mpi::AllGatherArrowBuffer(std::__1::shared_ptr<arrow::Buffer> const&, std::__1::shared_ptr<cylon::CylonContext> const&, std::__1::vector<std::__1::shared_ptr<arrow::Buffer>, std::__1::allocator<std::__1::shared_ptr<arrow::Buffer> > >&) in gather.cpp.o
      cylon::mpi::MpiAllgatherImpl::AllgatherBufferSize(int const*, int, int*) const in gather.cpp.o
      cylon::mpi::MpiTableBcastImpl::BcastBufferSizes(int*, int, int) const in bcast.cpp.o
      ...
  “_ompi_mpi_int64_t”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
  “_ompi_mpi_int8_t”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
  “_ompi_mpi_op_band”, referenced from:
      cylon::mpi::GetMPIOp(cylon::net::ReduceOp) in mpi_operations.cpp.o
  “_ompi_mpi_op_bor”, referenced from:
      cylon::mpi::GetMPIOp(cylon::net::ReduceOp) in mpi_operations.cpp.o
  “_ompi_mpi_op_land”, referenced from:
      cylon::mpi::GetMPIOp(cylon::net::ReduceOp) in mpi_operations.cpp.o
  “_ompi_mpi_op_lor”, referenced from:
      cylon::mpi::GetMPIOp(cylon::net::ReduceOp) in mpi_operations.cpp.o
  “_ompi_mpi_op_max”, referenced from:
      cylon::mpi::GetMPIOp(cylon::net::ReduceOp) in mpi_operations.cpp.o
  “_ompi_mpi_op_min”, referenced from:
      cylon::mpi::GetMPIOp(cylon::net::ReduceOp) in mpi_operations.cpp.o
  “_ompi_mpi_op_null”, referenced from:
      cylon::mpi::GetMPIOp(cylon::net::ReduceOp) in mpi_operations.cpp.o
      cylon::mpi::AllReduce(std::__1::shared_ptr<cylon::CylonContext> const&, void const*, void*, int, std::__1::shared_ptr<cylon::DataType> const&, cylon::net::ReduceOp) in mpi_operations.cpp.o
      cylon::mpi::MpiAllReduceImpl::AllReduceBuffer(void const*, void*, int, std::__1::shared_ptr<cylon::DataType> const&, cylon::net::ReduceOp) const in mpi_operations.cpp.o
  “_ompi_mpi_op_prod”, referenced from:
      cylon::mpi::GetMPIOp(cylon::net::ReduceOp) in mpi_operations.cpp.o
  “_ompi_mpi_op_sum”, referenced from:
      cylon::mpi::GetMPIOp(cylon::net::ReduceOp) in mpi_operations.cpp.o
  “_ompi_mpi_uint16_t”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
  “_ompi_mpi_uint32_t”, referenced from:
      cylon::mpi::GetMPIDataType(std::__1::shared_ptr<cylon::DataType> const&) in mpi_operations.cpp.o
  “_ompi_mpi_uint64_t”, referenced from:
      …
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
nirandaperera commented 1 year ago

@mstaylor could you please add the cmake output? Need to see if the MPI libs have been properly picked up

laszewsk commented 1 year ago

Is this M1or Intel? In other projects I made real bad experience with conda ... I do everything on M1 now from scratch, but that is a different project ...

mstaylor commented 1 year ago

M1

On Oct 6, 2022, at 12:20 PM, Gregor von Laszewski @.***> wrote:

 Is this M1or Intel? In other projects I made real bad experience with conda ... I do everything on M1 now from scratch, but that is a different project ...

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

laszewsk commented 1 year ago

have you done the conda install or an install from scratch?

mstaylor commented 1 year ago

I performed a fresh install from conda and experienced the same issue.

mstaylor commented 1 year ago

I'm able to use the debugger from my Intel MacBook Pro without any of these issues. This flag seems to fix the issue in silicon:

-DCMAKE_OSX_ARCHITECTURES=x86_64

See this: https://intellij-support.jetbrains.com/hc/en-us/community/posts/4406233940242-Using-CLion-on-Apple-Silicon-mac-to-target-x86-64-or-ARM-architecture

mstaylor commented 1 year ago

Conda install

On Oct 6, 2022, at 4:01 PM, Gregor von Laszewski @.***> wrote:

have you done the conda install or an install from scratch?

— Reply to this email directly, view it on GitHub https://github.com/cylondata/cylon/issues/625#issuecomment-1270612652, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA425C7HG6IKAF4QYYGEQXDWB4VX3ANCNFSM6AAAAAAQ6Y5MJI. You are receiving this because you were mentioned.