search

DUNE / larnd-sim

Simulation framework for a pixelated Liquid Argon TPC
Apache License 2.0
10 stars 26 forks source link
dune gpu lartpc neutrino physics

readme

larnd-sim

CI status Documentation DOI

larnd-sim

This software aims to simulate the light readout and the pixelated charge readout of a Liquid Argon Time Projection Chamber. It consists of a set of highly-parallelized algorithms implemented on the CUDA architecture using Numba.

Software documentation is available here. In addition, a paper about larnd-sim performance can be found here.

Overview

The software takes as input a dataset containing segments of deposited energy in the detector, generated with a Geant4 wrapper called edep-sim. The output of edep-sim is in the ROOT format and must be converted into HDF5 to be used by larnd-sim. For this purpose, we provide cli/dumpTree.py for non-beam events and 2x2_sim/run-convert2h5 /convert_edepsim_roottoh5.py (Note that this is in another repository, namely 2x2_sim) for beam events. The two versions will be merged in future.

larnd-sim simulates both the light and charge readout signals with ND-LAr like detector designs.

Installation

The package can be installed in this way:

git clone https://github.com/DUNE/larnd-sim.git
cd larnd-sim
pip install .

which should take care of installing the required dependencies. cupy installation might take a long time. You can considerably speed up the process by pre-installing cupy precompiled binaries, available here. The version will depend on the version of CUDA installed on your system. If you already have cupy installed in your environment which meets larnd-sim's requirements, you can execute export SKIP_CUPY_INSTALL=1 to skip cupy installation before running pip install ..

larnd-sim requires a CUDA-compatible GPU to function properly. To check if the GPU is setup properly and can talk to larnd-sim you can run:

>>> from numba.cuda import is_available
>>> is_available()

How to run a simulation

Command line interface

To run the simulation, simply execute the following:

simulate_pixels.py (--config CONFIG_KEYWORD) --input_filename INPUT_FILENAME --output_filename OUTPUT_FILENAME

Note that this is a new feature developed during the 2x2 MiniRun5 era (2023-12). It may not be available or functional before this commit in the develop branch.

Currently, CONFIG_KEYWORD supports module0, 2x2, 2x2_mod2mod_variation, 2x2_non_beam and ndlar, which cover the common use of larnd-sim. They are set in larndsim/config/config.yaml. Other configurations can be developed per request, and we welcome your PR.

If no argument given for --config, the simulation will use the defualt configuration 2x2_mod2od_variation.

Alternatively, the simulation can be run with explicit configurations listed in the command line such as (assuming it is run at the top level of the directory larnd-sim/.):

simulate_pixels.py \
--input_filename=INPUT_FOR_A_2x2_BEAM_EXAMPLE.h5 \
--output_filename=OUTPUT_FOR_A_2x2_BEAM_EXAMPLE.h5 \
--mod2mod_variation=False \
--pixel_layout=larndsim/pixel_layouts/multi_tile_layout-2.4.16.yaml\
--detector_properties=larndsim/detector_properties/2x2.yaml \
--response_file=larndsim/bin/response_44.npy \
--light_simulated=True \
--light_lut_filename=larndsim/bin/lightLUT.npz \
--light_det_noise_filename=larndsim/bin/light_noise_2x2_4mod_July2023.npy

Note that the default configuration is always active, so in order to control the configuration thoroughly and properly, please pass at least the list of arguments above.

You can also use a combination of the above two methods to configure the simulation. Taking the default configuration list and substitute some using the command line configuration. For example, here we simulate with four identical modules all with Module2-like LArPix geometry (3.8 mm pixel pitch etc.).

simulate_pixels.py --config 2x2 --pixel_layout=larndsim/pixel_layouts/multi_tile_layout-2.5.16.yaml\
--input_filename INPUT_FILENAME --output_filename OUTPUT_FILENAME

Input dataset

The input array can be created by converting edep-sim ROOT output files using the cli/dumpTree.py script (which is independent from the rest of the software and requires ROOT and Geant4 to be installed).

This script produces a bi-dimensional structured array saved in the HDF5 format, which is used as input for the simulation of the pixel response.

examples/lbnfSpillLAr.edep.h5 is an example of the input dataset. Otherwise you can find some example of 2x2 NuMI simulation input files on the 2x2_sim wiki.

Output

Detailed file data definition can be found in the 2x2_sim wiki.

Briefly, the larnd-sim output includes generator truth information, edep-sim/geant4 truth information, simulated charge detector output, charge backtracking information, detector propagated light truth information, simulated light detector output and light backtracking information.

The generator truth information includes mc_hdr and mc_stack which are reserved for neutrino interaction records. mc_hdr is the log for neutrino interactions, and mc_stack is the log for the initial-state and final-state particles from the neutrino interactions. This part is copied directly from the root converted h5 input file. If the upstream simulation does not run neutrino generator, rather cosmic or particle bomb simulation instead, then this information will not be presented in the output.

The edep-sim/geant4 truth information contains vertices (interaction-level or equivalent), trajectories (particle-level) and segments (segment of energy depositions). Primary trajectories should overlap with the final-state particles in mc_stack if it originates from neutrino generators. segments is the essential input to the simulation. This part should exist in the output regardless the simulation setup, and it is a direct copy of the corresponding part in the input with possible minor extensions such as event time (vertices['t_event']).

The simulated charge detector output is stored in packets which is compatible with the converted raw data from LArPix. A data packet could be considered as a point-like charge readout in time and the LArPix plane (anode). It encodes information of the readout charge, position and time. Other type of packets such as trigger, timestamp and sync are also available in the simulated packets dataset. A general discription can be found in the LArPix HDF5 documentation.

The charge backtracking information is namely mc_packets_assn which records the contribution of segments to each packet. It has the same size as the packets dataset, and the index are shared between packets and mc_packets_assn. Each packet is associated with backtracking information of event_ids, segment_ids and fraction regardless its packet type. However, only the data packets can have meaningful backtracking information filled. event_ids is a new feature implemented in this commit. It has a size of one and contains the true event_id. The length of segment_ids and fraction are the same and are set by fee.ASSOCIATION_COUNT_TO_STORE. The contributed segments are sorted by their fractional order and filled correspondingly for segment_ids and fraction. The total number of contribution considered in the calculation ofsegment fraction is set by fee.MAX_ADC_VALUES. It makes sense to set fee.MAX_ADC_VALUES >= fee.ASSOCIATION_COUNT_TO_STORE. Please see this Git Isuue for further details and discussion.

The detector propagated light truth information is saved as light_dat. If the module-by-module variation is turned on, the simulation is carried out for each module separately, and this information is stored for each module as light_dat/light_dat_module{Z} where Z is the module numbering. Otherwise, the information is recorded under light_dat/light_dat_allmodules. It has the shape of (#segments, #light readout channels), where the #segments and #light readout channels are within individual modules or the whole detector respectively. Each segment is labelled by segment_id. n_photons_det -- the number of photons which would detected by the SiPM, and t0_det -- the light arrival time on the SiPMs, are also provided in this dataset.

The simulated light detector output are light_trigand light_wvfm. Currently in larnd-sim, we have implemented two trigger mode: threshold (light.LIGHT_TRIG_MODE = 0) and beam (light.LIGHT_TRIG_MODE = 1) trigger that activate all light readout channels. See this Git Isuue for the discussion of light triggers in larnd-sim. light_trig has the shape of number of light triggers. In case of beam triggering mode, the number of light triggers is the same as the simulated events (beam spills) in the input. Therefore, not necessarily all light triggers correspond to charge and meaningful light signals. light_trig has attributes of op_channel, ts_s and ts_sync. op_channel records the triggered light readout channels. For light.LIGHT_TRIG_MODE = 0 and 1, op_channel stores all the light readout channel id. ts_s is the trigger time in seconds, and ts_sync is the trigger time in LArPix time ticks which considers the time sync, e.g PPS or LArPix clock rollover. light_wvfm has the shape of (#light triggers, #light readout channels, #light samples). The #light triggers is the same as in light_trig. For light.LIGHT_TRIG_MODE = 0 and 1, #light readout channels is the number of all the available channels in the entire detector. #light samples are determined by light.LIGHT_TRIG_WINDOW and light.LIGHT_DIGIT_SAMPLE_SPACING. For a light trigger and a readout channel light_wvfm gives a waveform as in the light readout data. The waveforms are in digitized ADC counts. For 2x2, and the cosmic data taking with Module 123, due to the setup, in order to get the ADC counts correspond to the light signal, the waveform values needs to be divided by 4. Note that the internal light simulation can have higher resolution before digitization if the light.LIGHT_TICK_SIZE is set to a smaller number.

The light backtracking information will be stored in light_wvfm_mc if light.MAX_MC_TRUTH_IDS is set to a non-zero value. light_wvfm_mc has the same shape as light_wvfm, (#light triggers, #light readout channels, #light samples). At one time sample of a light readout channel from one trigger, it stores segment_ids and pe_current with the length of light.MAX_MC_TRUTH_IDS.

For all light related datasets, as larnd-sim currently does not consider a complicated or realistic light readout channel mapping, their order is as the channel id which follows geometrical order sequentially from bottom to top, TPC-by-TPC.

Configuration files

Future developments

larnd-sim is fairly mature for the detector simulation that it is in vision to be. However, various developments are still needed to further improve the simulation. The Git Issues provides a peek into the wishlist. We welcome your contribution!

Currently, the development of larnd-sim is coordinated by Jaafar Chakrani (@jaafar-chakrani, LBNL), Kevin Wood (@krwood, LBNL), Yifan Chen (@YifanC, SLAC) and Matt Kramer (@mjkramer, LBNL), as part of the 2x2 simulation effort. The future simulation group of ND-LAr and/or its prototype will continue managing and maintaining larnd-sim. Please contact us if you have any suggestions, questions or concerns regarding larnd-sim.

Here, we would also like to acknowledge the initial authors of larnd-sim, Roberto Soleti (@soleti) and Peter Madigan (@peter-madigan). Thank you and many other contributors that have built larnd-sim.