This is in a branch based on control_replication commit e7e51ed1e22f51c8c616f926eefbebd17ddabcf7. I'm not 100% sure which variables matter, but here's what I know so far:
It replicates in a configuration with tracing, but without DCR or index launches
It does NOT replicate in the same configuration with index launches
As compared to other configurations I have run, this one is notable in that I enabled overdecomposition (10x as many tasks as GPUs)
It occurs starting at 4 nodes on Piz Daint
The relevant part of the backtrace looks like:
[2] Thread 8 (Thread 0x2aaaaac3c800 (LWP 29038) "circuit.noidx"):
[2] #0 0x00002aaaaf349217 in waitpid () from /lib64/libc.so.6
[2] #1 0x00002aaaaf2c676f in do_system () from /lib64/libc.so.6
[2] #2 0x00002aaaaccdd4bb in gasneti_bt_gdb () from /users/eslaught/regent-index-launch-sc21/language/circuit.run2_overdecompose_debug_2/libregent.so
[2] #3 0x00002aaaacce0dca in gasneti_print_backtrace () from /users/eslaught/regent-index-launch-sc21/language/circuit.run2_overdecompose_debug_2/libregent.so
[2] #4 0x00002aaaabf92979 in gasneti_defaultSignalHandler () from /users/eslaught/regent-index-launch-sc21/language/circuit.run2_overdecompose_debug_2/libregent.so
[2] #5 <signal handler called>
[2] #6 0x00002aaaac1d3645 in Legion::Internal::RegionTreePath::register_child (this=this@entry=0x2aab4024b4c8, depth=4294967295, color=33) at /users/eslaught/regent-index-launch-sc21/runtime/legion/legion_analysis.cc:19418
[2] #7 0x00002aaaac28b038 in Legion::Internal::RegionTreeForest::initialize_path (this=<optimized out>, child=0x2aab3cc88640, parent=0x2aab3cc81b80, path=...) at /users/eslaught/regent-index-launch-sc21/runtime/legion/region_tree.cc:5634
[2] #8 0x00002aaaac055a11 in Legion::Internal::VirtualCloseOp::initialize (this=this@entry=0x2aab4024b140, ctx=ctx@entry=0x2aab34a5b150, index=index@entry=1, req=..., target=0x2aab3440b560) at /users/eslaught/regent-index-launch-sc21/runtime/legion/legion_ops.cc:10212
[2] #9 0x00002aaaac11db5e in Legion::Internal::InnerContext::end_task (this=0x2aab34a5b150, res=<optimized out>, res_size=<optimized out>, owned=<optimized out>, deferred_result_instance=..., callback_functor=<optimized out>, result_kind=<optimized out>, freefunc=<optimized out>, metadataptr=<optimized out>, metadatasize=<optimized out>) at /users/eslaught/regent-index-launch-sc21/runtime/legion/legion_context.cc:10440
[2] #10 0x00002aaaac003884 in legion_task_postamble (runtime_=..., ctx_=..., retval=0x0, retsize=0) at /users/eslaught/regent-index-launch-sc21/runtime/legion/legion_c.cc:7727
[2] #11 0x000000000040d5ad in $__regent_task_init_piece_primary ()
[2] #12 0x00002aaaac820e51 in Realm::LocalTaskProcessor::execute_task (this=0x1613000, func_id=27, task_args=...) at /users/eslaught/regent-index-launch-sc21/runtime/realm/bytearray.inl:58
[2] #13 0x00002aaaac7165c3 in Realm::Task::execute_on_processor (this=0x2aab280ba950, p=...) at /users/eslaught/regent-index-launch-sc21/runtime/realm/tasks.cc:306
[2] #14 0x00002aaaac716656 in Realm::UserThreadTaskScheduler::execute_task (this=<optimized out>, task=<optimized out>) at /users/eslaught/regent-index-launch-sc21/runtime/realm/tasks.cc:1646
[2] #15 0x00002aaaac718cc9 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x10dc890) at /users/eslaught/regent-index-launch-sc21/runtime/realm/tasks.cc:1127
[2] #16 0x00002aaaac6feb3f in Realm::UserThread::uthread_entry () at /users/eslaught/regent-index-launch-sc21/runtime/realm/threads.cc:1337
[2] #17 0x00002aaaaf2ceca0 in ?? () from /lib64/libc.so.6
[2] #18 0x0000000000000000 in ?? ()
Reproduction Steps on Sapling
Copy my files:
cp -r /scratch2/eslaught/regent-index-launch-sc21 .
cd regent-index-launch-sc21
git stash
git reset --hard ec153e7d6da2cdeed0da69f5e8bb4bbe7b09e23b
git stash pop
This is in a branch based on control_replication commit e7e51ed1e22f51c8c616f926eefbebd17ddabcf7. I'm not 100% sure which variables matter, but here's what I know so far:
The relevant part of the backtrace looks like:
Reproduction Steps on Sapling
Copy my files:
Build:
Run:
Note that this command reproduces only about 50% of the time.