HPX does not work on OS X

[eschnett]

When I start an HPX program on OS X, then it aborts during startup with the error message

block_matrix(27778) malloc: *\ error for object 0x7fff76a41570: pointer being freed was not allocated

I am using gcc 4.8.1, with gcc, boost, and hwloc installed via MacPorts.

This is my backtrace:

* thread #1: tid = 0x4eeb90, 0x00007fff8b54ed46 libsystem_kernel.dylib`__kill + 10, queue = 'com.apple.main-thread, stop reason = signal SIGABRT
    frame #0: 0x00007fff8b54ed46 libsystem_kernel.dylib`__kill + 10
libsystem_kernel.dylib`__kill + 10:
-> 0x7fff8b54ed46:  jae    0x7fff8b54ed4d            ; __kill + 17
   0x7fff8b54ed48:  jmpq   0x7fff8b5504d4            ; cerror_nocancel
   0x7fff8b54ed4d:  ret    
   0x7fff8b54ed4e:  nop    
(lldb) bt
* thread #1: tid = 0x4eeb90, 0x00007fff8b54ed46 libsystem_kernel.dylib`__kill + 10, queue = 'com.apple.main-thread, stop reason = signal SIGABRT
    frame #0: 0x00007fff8b54ed46 libsystem_kernel.dylib`__kill + 10
    frame #1: 0x00007fff85950f83 libsystem_c.dylib`abort + 177
    frame #2: 0x00007fff85924989 libsystem_c.dylib`free + 392
    frame #3: 0x00000001077a4815 libstdc++.6.dylib`std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() + 21
    frame #4: 0x000000010000dc40 block_matrix`boost::filesystem::path::~path(this=0x00007fff5fbfe3c0) + 24 at path.hpp:55
    frame #5: 0x0000000101c44d29 libhpxd.1.dylib`hpx::util::detail::handle_generic_config_options(appname=string at 0x00007fff5fbfeac0, vm=0x00007fff5fbfebf0, desc_cfgfile=0x00007fff5fbfe830, ini=0x00007fff5fbff050, node=18446744073709551615) + 486 at parse_command_line.cpp:273
    frame #6: 0x0000000101c46120 libhpxd.1.dylib`hpx::util::parse_commandline(rtcfg=0x00007fff5fbff050, app_options=0x00007fff5fbff3e0, argc=1, argv=0x00007fff5fbff4b8, vm=0x00007fff5fbfebf0, node=18446744073709551615, error_mode=allow_unregistered, mode=runtime_mode_default, visible=0x0000000000000000, unregistered_options=0x0000000000000000) + 4183 at parse_command_line.cpp:562
    frame #7: 0x0000000101bcfe51 libhpxd.1.dylib`hpx::util::command_line_handling::call(this=0x00007fff5fbfefb0, desc_cmdline=0x00007fff5fbff3e0, argc=1, argv=0x00007fff5fbff4b8) + 203 at command_line_handling.cpp:635
    frame #8: 0x00000001015cbf3e libhpxd.1.dylib`hpx::run_or_start(f=0x00000001000f701b, desc_cmdline=0x00007fff5fbff3e0, argc=1, argv=0x00007fff5fbff4b8, ini_config=0x00007fff5fbff350, startup=0x00007fff5fbff470, shutdown=0x00007fff5fbff460, mode=runtime_mode_default, blocking=true)(boost::program_options::variables_map&), boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode, bool) + 177 at hpx_init.cpp:1064
    frame #9: 0x00000001015ccbe4 libhpxd.1.dylib`hpx::init(f=0x00000001000f701b, desc_cmdline=0x00007fff5fbff3e0, argc=1, argv=0x00007fff5fbff4b8, ini_config=0x00007fff5fbff350, startup=0x00007fff5fbff470, shutdown=0x00007fff5fbff460, mode=runtime_mode_default)(boost::program_options::variables_map&), boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode) + 85 at hpx_init.cpp:1173
    frame #10: 0x00000001000fbd99 block_matrix`hpx::init(f=0x00000001000f701b, desc_cmdline=0x00007fff5fbff3e0, argc=1, argv=0x00007fff5fbff4b8, startup=0x00007fff5fbff470, shutdown=0x00007fff5fbff460, mode=runtime_mode_default)(boost::program_options::variables_map&), boost::program_options::options_description const&, int, char**, hpx::util::function_nonser<void ()> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode) + 90 at hpx_init_impl.hpp:29
    frame #11: 0x00000001000fbe1b block_matrix`hpx::init(desc_cmdline=0x00007fff5fbff3e0, argc=1, argv=0x00007fff5fbff4b8, startup=0x00007fff5fbff470, shutdown=0x00007fff5fbff460, mode=runtime_mode_default)> const&, hpx::util::function_nonser<void ()> const&, hpx::runtime_mode) + 77 at hpx_init_impl.hpp:46
    frame #12: 0x00000001000f73c9 block_matrix`main(argc=1, argv=0x00007fff5fbff4b8) + 202 at main.cc:94
    frame #13: 0x00007fff8ae0e7e1 libdyld.dylib`start + 1

[eschnett]

I configure HPX via

cmake -DCMAKE_BUILD_TYPE=Debug -DHPX_HAVE_PARCELPORT_MPI=ON -DCMAKE_C_COMPILER=gcc \
    -DCMAKE_CXX_COMPILER=g++ -DCMAKE_FORTRAN_COMPILER=gfortran \
    -DMPI_CXX_COMPILER=openmpic++ -DMPI_C_COMPILER=openmpicc \
    -DMPI_FORTRAN_COMPILER=openmpif90 -DCMAKE_C_FLAGS="-Wno-unused-local-typedefs" \
    -DCMAKE_CXX_FLAGS="-Wno-unused-local-typedefs" -DBOOST_ROOT=/opt/local \
    -DHWLOC_ROOT=/opt/local -DCMAKE_INSTALL_PREFIX=$HOME/hpx $(pwd)/../hpx

and I configure my application via

cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ \
    -DCMAKE_C_FLAGS="-Wno-unused-local-typedefs" -DCMAKE_CXX_FLAGS="-Wno-unused-local-typedefs" \
    -G Ninja

[hkaiser]

Is that a new issue?

[eschnett]

No, this is the same issue that I discovered two weeks ago. I only just realized that I never opened a ticket for this.

[hkaiser]

BTW, HWLOC seems to have no effect on Mac as Mac OSX does not support defining thread affinities.

[eschnett]

Yes, this is not possible on OS X. Apple claims that their Grand Central Dispatch does a better job than setting affinities manually. Lucky for them, only Apple can actually test this claim.

However, hwloc can still be used to query system information, such as the number of cores.

[hkaiser]

We've had problems with tcmalloc on Mac OSX before. Could you please try with the system allocator instead?

[eschnett]

I am using the default, which is already the system malloc.

[hkaiser]

Bibek mentioned today that he was seeing similar problems when using boost built with clang with HPX which was built using gcc. Could that be the cause for your problems?

[eschnett]

Building HPX with the system compiler (clang 3.3svn) did not work; I receive a linker error that I didn't track down yet.

Building Boost myself with gcc worked out. My HPX application now starts up fine. There is an error at a later time (that I don't see on Linux), but this particular issue seems resolved.

[hkaiser]

Ok, thanks.