Test tests.unit.plugins.dist_matrixops.distributed.mpi.dist_constant_3_loc randomly fails

[stevenrbrandt]

The named test usually passes, sometimes times out, but sometimes fails. Sometimes the result is a simple failure, other times it's this...

[jovyan@3892a20b7fbb phylanx]$ (cd build && ctest -R tests.unit.plugins.dist_matrixops.distributed.mpi.dist_constant_3_loc )
Test project /home/jovyan/phylanx/build
    Start 182: tests.unit.plugins.dist_matrixops.distributed.mpi.dist_constant_3_loc
*** Error in `/home/jovyan/phylanx/build/bin/dist_constant_3_loc_test': malloc(): memory corruption: 0x00007f0b0c359e00 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7cbac)[0x7f0b730cbbac]
/lib64/libc.so.6(+0x8982b)[0x7f0b730d882b]
/lib64/libc.so.6(__libc_malloc+0x78)[0x7f0b730dbcb8]
/lib64/libstdc++.so.6(_Znwm+0x18)[0x7f0b73c103e8]
/usr/local/lib64/libhpxd.so.1(_ZN9__gnu_cxx13new_allocatorIN3hpx7threads20thread_data_stackfulEE8allocateEmPKv+0x4a)[0x7f0b76eb1bbc]
/usr/local/lib64/libhpxd.so.1(_ZN3hpx7threads20thread_data_stackful6createERNS0_16thread_init_dataEPvl+0x32)[0x7f0b76ead0e8]
/usr/local/lib64/libhpxd.so.1(+0xfb9ad7)[0x7f0b76ecaad7]
/usr/local/lib64/libhpxd.so.1(+0xfb55b8)[0x7f0b76ec65b8]
/usr/local/lib64/libhpxd.so.1(+0xfac5de)[0x7f0b76ebd5de]
/usr/local/lib64/libhpxd.so.1(+0xf9db47)[0x7f0b76eaeb47]
/usr/local/lib64/libhpxd.so.1(_ZN3hpx7threads6detail21scheduled_thread_poolINS0_8policies30local_priority_queue_schedulerISt5mutexNS3_13lockfree_fifoES6_NS3_13lockfree_lifoEEEE11create_workERNS0_16thread_init_dataERNS_10error_codeE+0x18d)[0x7f0b76f13a73]
/usr/local/lib64/libhpxd.so.1(+0x649d76)[0x7f0b7655ad76]
/usr/local/lib64/libhpxd.so.1(+0x66a4c6)[0x7f0b7657b4c6]
/usr/local/lib64/libhpxd.so.1(+0x8a1b02)[0x7f0b767b2b02]
/usr/local/lib64/libhpxd.so.1(+0x89cd45)[0x7f0b767add45]
/usr/local/lib64/libhpxd.so.1(+0x898d17)[0x7f0b767a9d17]
/usr/local/lib64/libhpxd.so.1(+0x894ad6)[0x7f0b767a5ad6]
/usr/local/lib64/libhpxd.so.1(+0x88f8d6)[0x7f0b767a08d6]
/usr/local/lib64/libhpxd.so.1(+0x888687)[0x7f0b76799687]
/usr/local/lib64/libhpxd.so.1(+0x8827ac)[0x7f0b767937ac]
/usr/local/lib64/libhpxd.so.1(+0x87c08d)[0x7f0b7678d08d]
/usr/local/lib64/libhpxd.so.1(_ZN3hpx4agas18addressing_service33send_refcnt_requests_non_blockingERSt11unique_lockINS_4lcos5local8spinlockEERNS_10error_codeE+0x54f)[0x7f0b7677260d]
/usr/local/lib64/libhpxd.so.1(_ZN3hpx4agas18addressing_service20send_refcnt_requestsERSt11unique_lockINS_4lcos5local8spinlockEERNS_10error_codeE+0x1ac)[0x7f0b76771fda]
/usr/local/lib64/libhpxd.so.1(_ZN3hpx4agas18addressing_service6decrefERKNS_6naming8gid_typeElRNS_10error_codeE+0x508)[0x7f0b7676c838]
/usr/local/lib64/libhpxd.so.1(_ZN3hpx4agas6decrefERKNS_6naming8gid_typeElRNS_10error_codeE+0x35)[0x7f0b7687139f]
/usr/local/lib64/libhpxd.so.1(+0xc3217d)[0x7f0b76b4317d]
/usr/local/lib64/libhpxd.so.1(_ZN3hpx6naming6detail19gid_managed_deleterEPNS1_12id_type_implE+0x29)[0x7f0b76b43419]
/usr/local/lib64/libhpxd.so.1(_ZN3hpx6naming6detail21intrusive_ptr_releaseEPNS1_12id_type_implE+0x45)[0x7f0b76b44e75]
/home/jovyan/phylanx/build/bin/dist_constant_3_loc_test[0x4870bb]
/home/jovyan/phylanx/build/bin/dist_constant_3_loc_test(_ZN3hpx6naming7id_typeD2Ev+0x18)[0x48414e]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x6ef62)[0x7f0b77a43f62]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x6e010)[0x7f0b77a43010]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x6cdf6)[0x7f0b77a41df6]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x76908)[0x7f0b77a4b908]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x7c554)[0x7f0b77a51554]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x7e375)[0x7f0b77a53375]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x7d9d8)[0x7f0b77a529d8]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x7c9b4)[0x7f0b77a519b4]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x44fcd)[0x7f0b77a19fcd]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x50f81)[0x7f0b77a25f81]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x6ec82)[0x7f0b77a43c82]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x7c5ed)[0x7f0b77a515ed]
/lib64/libc.so.6(__cxa_finalize+0x95)[0x7f0b7308a335]
/usr/local/lib64/libhpx_iostreamsd.so.1(+0x37de3)[0x7f0b77a0cde3]
======= Memory map: ========
00400000-004b7000 r-xp 00000000 fd:00 13894892                           /home/jovyan/phylanx/build/bin/dist_constant_3_loc_test
006b7000-006b8000 r--p 000b7000 fd:00 13894892                           /home/jovyan/phylanx/build/bin/dist_constant_3_loc_test
006b8000-006b9000 rw-p 000b8000 fd:00 13894892                           /home/jovyan/phylanx/build/bin/dist_constant_3_loc_test
006b9000-006bf000 rw-p 00000000 00:00 0
020b2000-0243c000 rw-p 00000000 00:00 0                                  [heap]
7f0b00000000-7f0b00021000 rw-p 00000000 00:00 0
7f0b00021000-7f0b04000000 ---p 00000000 00:00 0
7f0b04000000-7f0b04021000 rw-p 00000000 00:00 0
7f0b04021000-7f0b08000000 ---p 00000000 00:00 0
7f0b08000000-7f0b08021000 rw-p 00000000 00:00 0
7f0b08021000-7f0b0c000000 ---p 00000000 00:00 0
7f0b0c000000-7f0b0c361000 rw-p 00000000 00:00 0
7f0b0c361000-7f0b10000000 ---p 00000000 00:00 0
7f0b10000000-7f0b10021000 rw-p 00000000 00:00 0
7f0b10021000-7f0b14000000 ---p 00000000 00:00 0
7f0b14000000-7f0b14021000 rw-p 00000000 00:00 0
7f0b14021000-7f0b18000000 ---p 00000000 00:00 0
7f0b18000000-7f0b18028000 rw-p 00000000 00:00 0
7f0b18028000-7f0b1c000000 ---p 00000000 00:00 0
7f0b1c000000-7f0b1c021000 rw-p 00000000 00:00 0
7f0b1c021000-7f0b20000000 ---p 00000000 00:00 0
7f0b20000000-7f0b20021000 rw-p 00000000 00:00 0
7f0b20021000-7f0b24000000 ---p 00000000 00:00 0
7f0b24000000-7f0b24021000 rw-p 00000000 00:00 0
7f0b24021000-7f0b28000000 ---p 00000000 00:00 0
7f0b28000000-7f0b28021000 rw-p 00000000 00:00 0
7f0b28021000-7f0b2c000000 ---p 00000000 00:00 0
7f0b2de82000-7f0b2de83000 ---p 00000000 00:00 0
7f0b2de83000-7f0b2dea3000 rwxp 00000000 00:00 0
7f0b2dea3000-7f0b2dea4000 ---p 00000000 00:00 0
7f0b2dea4000-7f0b2dec4000 rwxp 00000000 00:00 0
7f0b2dec4000-7f0b2dec5000 ---p 00000000 00:00 0
7f0b2dec5000-7f0b2dee5000 rwxp 00000000 00:00 0
7f0b2dee5000-7f0b2dee6000 ---p 00000000 00:00 0
7f0b2dee6000-7f0b2df06000 rwxp 00000000 00:00 0
7f0b2df06000-7f0b2df07000 ---p 00000000 00:00 0
7f0b2df07000-7f0b2df27000 rwxp 00000000 00:00 0
7f0b2df27000-7f0b2df28000 ---p 00000000 00:00 0
7f0b2df28000-7f0b2df48000 rwxp 00000000 00:00 0
7f0b2df48000-7f0b2df49000 ---p 00000000 00:00 0
7f0b2df49000-7f0b2df69000 rwxp 00000000 00:00 0
7f0b2df69000-7f0b2df6a000 ---p 00000000 00:00 0
7f0b2df6a000-7f0b2df8a000 rwxp 00000000 00:00 0
7f0b2df8a000-7f0b2df8b000 ---p 00000000 00:00 0
7f0b2df8b000-7f0b2dfab000 rwxp 00000000 00:00 0
7f0b2dfab000-7f0b2dfac000 ---p 00000000 00:00 0
7f0b2dfac000-7f0b2dfcc000 rwxp 00000000 00:00 0
7f0b2dfcc000-7f0b2dfcd000 ---p 00000000 00:00 0
7f0b2dfcd000-7f0b2dfed000 rwxp 00000000 00:00 0
7f0b2dfed000-7f0b2dfee000 ---p 00000000 00:00 0
7f0b2dfee000-7f0b2e7ee000 rw-p 00000000 00:00 0
7f0b2e7ee000-7f0b2e7ef000 ---p 00000000 00:00 0
7f0b2e7ef000-7f0b2e80f000 rwxp 00000000 00:00 0
7f0b2e80f000-7f0b2e810000 ---p 00000000 00:00 0
7f0b2e810000-7f0b2e830000 rwxp 00000000 00:00 0
7f0b2e830000-7f0b2e831000 ---p 00000000 00:00 0
7f0b2e831000-7f0b2e851000 rwxp 00000000 00:00 0
7f0b2e851000-7f0b2e852000 ---p 00000000 00:00 0
7f0b2e852000-7f0b2ea52000 rwxp 00000000 00:00 0
7f0b2ea52000-7f0b2ea53000 ---p 00000000 00:00 0
7f0b2ea53000-7f0b2ea73000 rwxp 00000000 00:00 0
7f0b2ea73000-7f0b2ea74000 ---p 00000000 00:00 0
7f0b2ea74000-7f0b2ea94000 rwxp 00000000 00:00 0
7f0b2ea94000-7f0b2ea95000 ---p 00000000 00:00 0
7f0b2ea95000-7f0b2eab5000 rwxp 00000000 00:00 0
7f0b2eab5000-7f0b2eab6000 ---p 00000000 00:00 0
7f0b2eab6000-7f0b2ead6000 rwxp 00000000 00:00 0
7f0b2ead6000-7f0b2ead7000 ---p 00000000 00:00 0
7f0b2ead7000-7f0b2eaf7000 rwxp 00000000 00:00 0
7f0b2eaf7000-7f0b2eaf8000 ---p 00000000 00:00 0
7f0b2eaf8000-7f0b2eb18000 rwxp 00000000 00:00 0
7f0b2eb18000-7f0b2eb19000 ---p 00000000 00:00 0
7f0b2eb19000-7f0b2eb39000 rwxp 00000000 00:00 0
7f0b2eb39000-7f0b2eb3a000 ---p 00000000 00:00 0
7f0b2eb3a000-7f0b2eb5a000 rwxp 00000000 00:00 0
7f0b2eb5a000-7f0b2eb5b000 ---p 00000000 00:00 0
7f0b2eb5b000-7f0b2eb7b000 rwxp 00000000 00:00 0
7f0b2eb7b000-7f0b2eb7c000 ---p 00000000 00:00 0
7f0b2eb7c000-7f0b2eb9c000 rwxp 00000000 00:00 0
7f0b2eb9c000-7f0b2eb9d000 ---p 00000000 00:00 0
7f0b2eb9d000-7f0b2ebbd000 rwxp 00000000 00:00 0
7f0b2ebbd000-7f0b2ebbe000 ---p 00000000 00:00 0
7f0b2ebbe000-7f0b2ebde000 rwxp 00000000 00:00 0
7f0b2ebde000-7f0b2ebdf000 ---p 00000000 00:00 0
7f0b2ebdf000-7f0b2ebff000 rwxp 00000000 00:00 0
7f0b2ebff000-7f0b2ec00000 ---p 00000000 00:00 0
7f0b2ec00000-7f0b2ec20000 rwxp 00000000 00:00 0
7f0b2ec20000-7f0b2ec21000 ---p 00000000 00:00 0
7f0b2ec21000-7f0b2ec41000 rwxp 00000000 00:00 0
7f0b2ec41000-7f0b2ec42000 ---p 00000000 00:00 0
7f0b2ec42000-7f0b2ec62000 rwxp 00000000 00:00 0
7f0b2ec62000-7f0b2ec63000 ---p 00000000 00:00 0
7f0b2ec63000-7f0b2ec83000 rwxp 00000000 00:00 0
7f0b2ec83000-7f0b2ec84000 ---p 00000000 00:00 0
7f0b2ec84000-7f0b2eca4000 rwxp 00000000 00:00 0
7f0b2eca4000-7f0b2eca5000 ---p 00000000 00:00 0
7f0b2eca5000-7f0b2ecc5000 rwxp 00000000 00:00 0
7f0b2ecc5000-7f0b2ecc6000 ---p 00000000 00:00 0
7f0b2ecc6000-7f0b2ece6000 rwxp 00000000 00:00 0
7f0b2ece6000-7f0b2ece7000 ---p 00000000 00:00 0
7f0b2ece7000-7f0b2ed07000 rwxp 00000000 00:00 0
7f0b2ed07000-7f0b2ed08000 ---p 00000000 00:00 0
7f0b2ed08000-7f0b2ed28000 rwxp 00000000 00:00 0
7f0b2ed28000-7f0b2ed29000 ---p 00000000 00:00 0
7f0b2ed29000-7f0b2ed49000 rwxp 00000000 00:00 0
7f0b2ed49000-7f0b2ed4a000 ---p 00000000 00:00 0
7f0b2ed4a000-7f0b2ed6a000 rwxp 00000000 00:00 0
7f0b2ed6a000-7f0b2ed6b000 ---p 00000000 00:00 0
7f0b2ed6b000-7f0b2ed8b000 rwxp 00000000 00:00 0
7f0b2ed8b000-7f0b2ed8c000 ---p 00000000 00:00 0
7f0b2ed8c000-7f0b2edac000 rwxp 00000000 00:00 0
7f0b2edac000-7f0b2edad000 ---p 00000000 00:00 0
7f0b2edad000-7f0b2edcd000 rwxp 00000000 00:00 0
7f0b2edcd000-7f0b2edce000 ---p 00000000 00:00 0
7f0b2edce000-7f0b2edee000 rwxp 00000000 00:00 0
7f0b2edee000-7f0b2edef000 ---p 00000000 00:00 0
7f0b2edef000-7f0b2ee0f000 rwxp 00000000 00:00 0
7f0b2ee0f000-7f0b2ee10000 ---p 00000000 00:00 0
7f0b2ee10000-7f0b2ee30000 rwxp 00000000 00:00 0
7f0b2ee30000-7f0b2ee31000 ---p 00000000 00:00 0
7f0b2ee31000-7f0b2ee51000 rwxp 00000000 00:00 0
7f0b2ee51000-7f0b2ee52000 ---p 00000000 00:00 0
7f0b2ee52000-7f0b2ee72000 rwxp 00000000 00:00 0
7f0b2ee72000-7f0b2ee73000 ---p 00000000 00:00 0
7f0b2ee73000-7f0b2ee93000 rwxp 00000000 00:00 0
7f0b2ee93000-7f0b2ee94000 ---p 00000000 00:00 0
7f0b2ee94000-7f0b2eeb4000 rwxp 00000000 00:00 0
7f0b2eeb4000-7f0b2eeb5000 ---p 00000000 00:00 0
7f0b2eeb5000-7f0b2eed5000 rwxp 00000000 00:00 0
7f0b2eed5000-7f0b2eed6000 ---p 00000000 00:00 0
7f0b2eed6000-7f0b2eef6000 rwxp 00000000 00:00 0
7f0b2eef6000-7f0b2eef7000 ---p 00000000 00:00 0
7f0b2eef7000-7f0b2ef17000 rwxp 00000000 00:00 0
7f0b2ef17000-7f0b2ef18000 ---p 00000000 00:00 0
7f0b2ef18000-7f0b2ef38000 rwxp 00000000 00:00 0
7f0b2ef38000-7f0b2ef39000 ---p 00000000 00:00 0
7f0b2ef39000-7f0b2ef59000 rwxp 00000000 00:00 0
7f0b2ef59000-7f0b2ef5a000 ---p 00000000 00:00 0
7f0b2ef5a000-7f0b2ef7a000 rwxp 00000000 00:00 0
7f0b2ef7a000-7f0b2ef7b000 ---p 00000000 00:00 0
7f0b2ef7b000-7f0b2ef9b000 rwxp 00000000 00:00 0
7f0b2ef9b000-7f0b2ef9c000 ---p 00000000 00:00 0
7f0b2ef9c000-7f0b2efbc000 rwxp 00000000 00:00 0
7f0b2efbc000-7f0b2efbd000 ---p 00000000 00:00 0
7f0b2efbd000-7f0b2efdd000 rwxp 00000000 00:00 0
7f0b2efdd000-7f0b2efde000 ---p 00000000 00:00 0
7f0b2efde000-7f0b2effe000 rwxp 00000000 00:00 0
7f0b2effe000-7f0b2efff000 ---p 00000000 00:00 0
7f0b2efff000-7f0b2f7ff000 rw-p 00000000 00:00 0
7f0b2f7ff000-7f0b2f800000 ---p 00000000 00:00 0
7f0b2f800000-7f0b30000000 rw-p 00000000 00:00 0
7f0b30000000-7f0b30021000 rw-p 00000000 00:00 0
7f0b30021000-7f0b34000000 ---p 00000000 00:00 0
7f0b34015000-7f0b34016000 ---p 00000000 00:00 0
7f0b34016000-7f0b34036000 rwxp 00000000 00:00 0
7f0b34036000-7f0b34037000 ---p 00000000 00:00 0
7f0b34037000-7f0b34057000 rwxp 00000000 00:00 0
7f0b34057000-7f0b34058000 ---p 00000000 00:00 0
7f0b34058000-7f0b34078000 rwxp 00000000 00:00 0
7f0b34078000-7f0b34079000 ---p 00000000 00:00 0
7f0b34079000-7f0b34099000 rwxp 00000000 00:00 0
7f0b34099000-7f0b3409a000 ---p 00000000 00:00 0
7f0b3409a000-7f0b340ba000 rwxp 00000000 00:00 0
7f0b340ba000-7f0b340bb000 ---p 00000000 00:00 0
7f0b340bb000-7f0b340db000 rwxp 00000000 00:00 0
7f0b340db000-7f0b340dc000 ---p 00000000 00:00 0
7f0b340dc000-7f0b340fc000 rwxp 00000000 00:00 0
7f0b340fc000-7f0b340fd000 ---p 00000000 00:00 0
7f0b340fd000-7f0b3411d000 rwxp 00000000 00:00 0
7f0b3411d000-7f0b3411e000 ---p 00000000 00:00 0
7f0b3411e000-7f0b3413e000 rwxp 00000000 00:00 0
7f0b3413e000-7f0b3413f000 ---p 00000000 00:00 0
7f0b3413f000-7f0b3415f000 rwxp 00000000 00:00 0
7f0b3415f000-7f0b34160000 ---p 00000000 00:00 0
7f0b34160000-7f0b34180000 rwxp 00000000 00:00 0
7f0b34180000-7f0b34181000 ---p 00000000 00:00 0
7f0b34181000-7f0b341a1000 rwxp 00000000 00:00 0
7f0b341a1000-7f0b341a2000 ---p 00000000 00:00 0
7f0b341a2000-7f0b341c2000 rwxp 00000000 00:00 0
7f0b341c2000-7f0b341c3000 ---p 00000000 00:00 0
7f0b341c3000-7f0b341e3000 rwxp 00000000 00:00 0

[khuck]

@hkaiser @stevenrbrandt I don't know if this is related, but I was able to reproduce the random MPI/TCP errors in the phylanx tests. HPX is throwing a failed assertion error when the runtime requested by hpx::get_runtime_distributed() or hpx::get_runtime() returns nullptr. Is there a race condition during shutdown?:

{stack-trace}: 19 frames:
0x7f5c3588825b  : hpx::detail::custom_exception_info(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0xdd] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c356f4dbc  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xd1ddbc) [0x7f5c356f4dbc] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c356cc8bb  : std::__exception_ptr::exception_ptr hpx::detail::construct_custom_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x11f] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c356caf67  : std::__exception_ptr::exception_ptr hpx::detail::get_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x77] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c35894ee9  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xebdee9) [0x7f5c35894ee9] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c355e5ad2  : hpx::assertion::detail::handle_assert(hpx::assertion::source_location const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0xee] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c35528567  : hpx::get_runtime_distributed() [0x60] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c35529909  : hpx::naming::get_agas_client() [0x9] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c354f8440  : hpx::parcelset::parcel::determine_lva() [0x18] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c354f8812  : hpx::parcelset::parcel::load_schedule(hpx::serialization::input_archive&, unsigned long, bool&) [0xd6] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c35576312  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xb9f312) [0x7f5c35576312] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c355731c7  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xb9c1c7) [0x7f5c355731c7] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c3556ff4c  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xb98f4c) [0x7f5c3556ff4c] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c3556c10a  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xb9510a) [0x7f5c3556c10a] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c3556bf91  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xb94f91) [0x7f5c3556bf91] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c3556bdd7  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xb94dd7) [0x7f5c3556bdd7] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c3556bc59  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xb94c59) [0x7f5c3556bc59] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c355685ba  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xb915ba) [0x7f5c355685ba] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f5c35566210  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xb8f210) [0x7f5c35566210] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
{locality-id}: 4294967295
{process-id}: 26033
{os-thread}: <unknown>
{thread-description}: <unknown>
{state}: not running
{auxinfo}: 
{file}: /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/src/hpx/src/runtime_distributed.cpp
{line}: 1807
{function}: hpx::runtime_distributed& hpx::get_runtime_distributed()
{what}: Assertion 'get_runtime_distributed_ptr() != nullptr' failed: HPX(assertion_failure)

[khuck]

I also get errors like:

{stack-trace}: 19 frames:
0x7f1da9fc925b  : hpx::detail::custom_exception_info(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0xdd] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e35dbc  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xd1ddbc) [0x7f1da9e35dbc] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e0d8bb  : std::__exception_ptr::exception_ptr hpx::detail::construct_custom_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x11f] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e0bf67  : std::__exception_ptr::exception_ptr hpx::detail::get_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x77] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e0c751  : void hpx::detail::throw_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) [0x91] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e1585f  : hpx::detail::throw_exception(hpx::error, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) [0x85] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e15af8  : hpx::detail::throws_if(hpx::error_code&, hpx::error, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) [0x4c] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1daa0734d2  : hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::create_work(hpx::threads::thread_init_data&, hpx::error_code&) [0x10e] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da97383d0  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0x6203d0) [0x7f1da97383d0] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9738403  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0x620403) [0x7f1da9738403] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9af8ad6  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0x9e0ad6) [0x7f1da9af8ad6] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9af32bd  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0x9db2bd) [0x7f1da9af32bd] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9aee7c1  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0x9d67c1) [0x7f1da9aee7c1] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9ae9b6f  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0x9d1b6f) [0x7f1da9ae9b6f] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9ae3f8c  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0x9cbf8c) [0x7f1da9ae3f8c] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9adec18  : void hpx::actions::transfer_continuation_action<hpx::agas::server::symbol_namespace::on_event_action>::schedule_thread<0ul, 1ul, 2ul>(hpx::util::pack_c<unsigned long, 0ul, 1ul, 2ul>, hpx::naming::gid_type const&, unsigned long, int, unsigned long) [0x234] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9adaa41  : hpx::actions::transfer_continuation_action<hpx::agas::server::symbol_namespace::on_event_action>::schedule_thread(hpx::naming::gid_type const&, unsigned long, int, unsigned long) [0x3f] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9adab54  : hpx::actions::transfer_continuation_action<hpx::agas::server::symbol_namespace::on_event_action>::load_schedule(hpx::serialization::input_archive&, hpx::naming::gid_type&&, unsigned long, int, unsigned long, bool&) [0x8a] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9c39904  : hpx::parcelset::parcel::load_schedule(hpx::serialization::input_archive&, unsigned long, bool&) [0x1c8] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
{locality-id}: 0
{hostname}: [ (tcp:127.0.0.1:7910) ]
{process-id}: 27061
{os-thread}: locality#0/parcel-thread-tcp#0
{thread-description}: <unknown>
{state}: state_pre_startup
{auxinfo}:
{file}: /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/src/hpx/libs/thread_pools/include/hpx/thread_pools/scheduled_thread_pool_impl.hpp
{line}: 602
{function}: thread_pool<Scheduler>::create_work
{what}: invalid state: thread pool is not running: HPX(invalid_status)

and

{stack-trace}: 19 frames:
0x7f1da9fc925b  : hpx::detail::custom_exception_info(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0xdd] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e35dbc  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xd1ddbc) [0x7f1da9e35dbc] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e0d8bb  : std::__exception_ptr::exception_ptr hpx::detail::construct_custom_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x11f] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e0bf67  : std::__exception_ptr::exception_ptr hpx::detail::get_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [0x77] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e0c751  : void hpx::detail::throw_exception<hpx::exception>(hpx::exception const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) [0x91] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9e1585f  : hpx::detail::throw_exception(hpx::error, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) [0x85] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x7f1da9cbebc2  : /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1(+0xba6bc2) [0x7f1da9cbebc2] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/hpx-Debug/lib/libhpxd.so.1
0x48783e        : hpx::serialization::input_archive::load_binary(void*, unsigned long) [0x42] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/bin/retile_3_loc_test
0x7f1db02ee6c3  : hpx::serialization::basic_archive<hpx::serialization::input_archive>::load_binary(void*, unsigned long) [0x2b] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02ee40e  : void hpx::serialization::load_binary<hpx::serialization::input_archive>(hpx::serialization::input_archive&, void*, unsigned long) [0x2b] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02ee12e  : void hpx::serialization::serialize<char, std::char_traits<char>, std::allocator<char> >(hpx::serialization::input_archive&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned int) [0x95] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02eddff  : void hpx::serialization::serialize_non_intrusive<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, void>::call<hpx::serialization::input_archive>(hpx::serialization::input_archive&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned int) [0x42] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02edb4d  : void hpx::serialization::access::serialize_dispatcher<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::non_intrusive::call<hpx::serialization::input_archive>(hpx::serialization::input_archive&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned int) [0x2b] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02ed87e  : void hpx::serialization::access::serialize<hpx::serialization::input_archive, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(hpx::serialization::input_archive&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned int) [0x2b] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02ed528  : void hpx::serialization::input_archive::load_nonintrusively_polymorphic<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::integral_constant<bool, false>) [0x28] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02ecf38  : void hpx::serialization::input_archive::load_bitwise<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::integral_constant<bool, false>) [0x28] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02ec600  : std::enable_if<(!std::is_integral<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::value)&&(!std::is_enum<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::value), void>::type hpx::serialization::input_archive::load<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) [0x28] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02eaded  : void hpx::serialization::input_archive::invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) [0x23] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
0x7f1db02e964b  : void hpx::serialization::basic_archive<hpx::serialization::input_archive>::invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) [0x23] in /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/build-delphi-x86_64-Linux-gcc/phylanx-Debug/lib/libhpx_phylanxd.so.0
{locality-id}: 0
{hostname}: [ (tcp:127.0.0.1:7910) ]
{process-id}: 27061
{os-thread}: locality#0/parcel-thread-tcp#1
{thread-description}: <unknown>
{state}: state_pre_startup
{auxinfo}:
{file}: /var/lib/buildbot/workers/phylanx/x86_64-gcc7-debug/build/tools/buildbot/src/hpx/libs/serialization/include/hpx/serialization/input_container.hpp
{line}: 113
{function}: input_container::load_binary
{what}: archive data bstream is too short: HPX(serialization_error)

and

  what():  <unknown>: HPX(network_error)