pietrodigangi
commented
3 years ago thanks @WenzDaniel, this sounds like a good idea! let's also discuss it in the next MC meeting.
Open WenzDaniel opened 3 years ago
pietrodigangi
commented
3 years ago thanks @WenzDaniel, this sounds like a good idea! let's also discuss it in the next MC meeting.
JoranAngevaare
commented
3 years ago Hi Daniel, nice idea! Hope that it's easy to get it all working.
Just one side note before you put a lot of effort in getting it working multiprocessing, let me point out that it's not always the holy grail. We use it on the eventbuilders but there we hyperfinetuned the settings (timeout, max-messages, n-cores) to get it working with such great performance. Generally the lazy mode makes things easier, the multiprocessing trace-backs are not the most fun to deal with.
In principle, it's easy to parallelize wfsim by just creating new runs. Since you don't have to have ints as runs you can do 1000_a, 1000_b and just have multiple jobs running on them. Data has to be sorted by time within a run, but not per se among runs.
petergaemers
commented
3 years ago Hi Joran. Actually most of this is not just for multiprocessing. It's needed to have good communication between epix, wfsim and straxen. So we need to do most of this if not all anyway. And the things needed are pretty straightforward, we don't need to do anything complicated
Issue:
WFSim is computationally the heaviest part in our full chain simulation. Therefore, it would be nice if we could utilize strax's Mailbox-System for parallel processing. However, this can be a big hurtle since we have to switch from a eventid based ordering scheme used by GEANT4 to a time based one which is used by strax/en (This issue has to be overcome anyhow since we want to use strax/straxen in our analyses.). Hence, in order to use multiprocessing with WFSim, EPIX has to provide non-overlapping, time sorted and time chunked data. I discussed this idea already with @HenningSE, @petergaemers and @ramirezdiego and would like to use this issue to present the idea in more detail. The changes should be minior, mainly in EPIX and fast to be implemented. Further, I would like to ask people to comment in case I overlooked any obstacles.
Proposal:
The main concept is sketch in the subsequent figure.
Step 1. GEANT4: GEANT4 is the starting point of our full chain simulation and computationally the second heaviest part of the entire chain. In my current proposal we would exploit the already recently implemented features like a continuous running eventid and use multiple jobs running in parallel to simulate our data. Each job writes its results into a file in a single directory. The filename would have to follow some convention we have to agree on e.g. like sim_runid_000001.root....
Step 2. EPIXReader:
After GEANT4 we use EPIX to define interactions and timestamps which are passed as instructions to WFSim. In previous proposals for our full-chain-simulation approach, EPIX runs together with GEANT4 in independent parallel jobs. However, I think we should run it as a single serial job which loops over all created GEANT4 files.
EPIX poses our best chance to transform our data from an eventid based to a time based ordering scheme. It is very fast compared to the other tools mentioned. It takes about ~2-3 minutes for 15k neutrons events. Hence, it would not create a new bottleneck if it is not parallelized. And, we already define timestamps in EPIX which are passed as instructions to WFSim.
The changes would look like the following: We move EPIX into a strax-plugin which has the task to provide well defined, non-overlapping chunks of data. This is similar to the DAQReader for regular data-taking ("EPIXReader"). This can only be ensured if EPIXReader loops over our data in a serial way and postpone events (or S2s of an event) into a future chunk if needed. The chunks can be created in two different ways. Either based on an event rate or well separated events for a clean simulation. Further should each chunk always contain about 5k events to ensure a not too large I/O-overhead. Depending on the user specified rate or creation method the chunk size may vary from simulation to simulation, but this should not be of any problem.
However, there are two obstacle which we have to overcome. On the other hand this issue would occur the either way. Due to after pulse simulation in WFSim, e.g. photo-ionization signals, signals can be produced far beyond the chunk boundaries if computed in a naive way. Hence I would propose, that the chunk-endtime is always computed such that we can accommodated afterpulses up to a max delay of 4 drift lengths after the original interaction (can also either reduce or increase this number if needed). Afterpulses beyond that time are truncated. For the very same reason it may be required in a few cases to extend the chunk-boundary a bit. This introduces a small but negligible error on the event rate. E.g. if we have a source with a rate of 50 Bq we have a chunksize of 100 s. If we are unlucky and the last S2 is placed at a time of 100s we have to extend the chunk by 4 x max drift time.
Step 3.: WFSim would remain more or less unchanged, I think. We just need to add the 4 x max drift truncation and switch to multi-processing.
Advantages:
Disdvantage:
Peter started the transition to a strax-plugin already: https://github.com/XENONnT/epix/pull/7, but we have to make the changes on the timestamp creation first.
ToDo:
In the following I list the required changes. Step 2. changes can be done at a later time once we verified the full chain simulation is working.
Epix:
WFSim:
Both: