Open sethrj opened 4 months ago
This is going to be troublesome for the different ways that we execute across threads. It's easy if we're doing OpenMP and know that all states are going to be starting and stopping at a synchronization point, it's easy to send a vector of state references when each thread is finished but the states are still allocated. However, if we're running through Geant4 MT, the "EndOfRunAction" will be called individually on each thread and then on the "master" thread. But we have to deallocate the state on the original thread, which creates an ordering issue.
Perhaps instead of trying to make our destructor ordering work with Geant4's threading model, we add special cases for anything that has to add Geant4 objects:
Now that we have a LocalTransporter
we could have it manage shared pointers to the hit processors, and then give weak pointers (for safety) + raw pointers (for performance, knowing that since the hit processes are only "shared" within a single thread we don't have to use locking) to the hit manager.
We also need to add an aux state vector interface to the StepInterface::process_steps
so that step processors can have stateful data without collection mirrors.
So the order of this will be:
LocalTransporter
for deallocating geant4 geometry states on thread (?)
The
StreamStore
is no longer needed since #1278, except that diagnostic classes still use it for doing a reduction over multiple states for output. I think we need to break apart this functionality since we really don't want the params to be mutable and keep access to multiple states.EndRunActionInterface
that takes a the core params, a core state, and aSpan<CoreState<M> const*>
of all states for performing a reduction in a multithread context. The action itself should know whether to do a global reduction or otherwise: probably it should always reduce toStreamId{0}
. The action should check thatstate.stream_id() < all_states.size() && all_states[state.stream_id().get()] == &state
. We probably want to add an MPI communicator to the core params so that we can perform reductions with dynamic parallelism.max_streams
parameter.begin_run
action is taken when aStepper
is created, which (for celer-sim) may happen separately in warmup and inside the parallel loop, or (for theaccel
interface) will happen during the run manager'sBeginOfRunAction
call. We should make sure theend_run
action is executed in parallel...