spill simulation

[krwood]

Adds new simulation script and functionality for simulating spills from neutrino beams

[peter-madigan]

@krwood I think keeping track of two primary simulation scripts is going to be kind of a nightmare. Since the differences are fairly limited (see below), it seems like it might make more sense to try and work these into configuration parameters. I think the end goal for simulate_pixels.py should be to be an entry point into larndsim, and have most of the simulation settings configurable within the yaml files that it consumes. There's also probably a clever way to create command line arguments that take precedence over the yaml file (handy, but is probably a low priority for the moment).

I think you could do something like the following:

1. add a file larndsim/consts/sim.py
2. within that file set defaults for BATCH_SIZE, EVENT_BATCH_SIZE, WRITE_BATCH_SIZE
3. create a set_simulation_properties function along the lines of larndsim.detector.set_detector_properties, to update these values from the detector properties yaml
4. create additional variables for GENERATE_EVENT_TIMESTAMPS (to determine if you use fee.gen_event_timestamps or bypass it), SKIP_NON_ACTIVE_TRACKS (to determine if you want to bypass the active_volume.* stuff), USE_INDIV_CHANNEL_THRESHOLDS (to determine if you want to bypass the single channel thresholds)
5. grab these configuration values from larndsim.consts.sim in simulate_pixels.py

There are also lots of configuration values (e.g. all the gain/vref/vcm/etc in fee.py) that should also be made configurable from the yaml file, but that might be less urgent.

$ diff -y --left-column --suppress-common-lines -W 150 cli/simulate_pixels.py cli/simulate_pixels_spills.py

from numba.cuda import device_array, to_device                | from numba.cuda import device_array, to_device, synchronize
                                      > # krw fixing seed to test code changes
#BATCH_SIZE = 4000 # track segments                   | BATCH_SIZE = 10000 # track segments
BATCH_SIZE = 4000 # track segments                    | EVENT_BATCH_SIZE = 1 # tpcs
EVENT_BATCH_SIZE = 2 # tpcs                       | WRITE_BATCH_SIZE = 1 # batches
#WRITE_BATCH_SIZE = 1000 # batches                    <
WRITE_BATCH_SIZE = 1000 # batches                     <
    print("Skipping non-active volumes..." , end="")              |     #print("Skipping non-active volumes..." , end="")
    start_mask = time()                           |     #start_mask = time()
    active_tracks = active_volume.select_active_volume(tracks, detecto    |     #active_tracks = active_volume.select_active_volume(tracks, detect
    tracks = tracks[active_tracks]                    |     #track_len_before_mask = len(tracks)
    segment_ids = segment_ids[active_tracks]                  |     #tracks = tracks[active_tracks]
    end_mask = time()                             |     #segment_ids = segment_ids[active_tracks]
    print(f" {end_mask-start_mask:.2f} s")                |     #n_tracks_cut = track_len_before_mask - len(tracks)
                                      >     #end_mask = time()
                                      >     #print(f" {end_mask-start_mask:.2f} s")
                                      >     #print('Active volume cut ',n_tracks_cut,'tracks.')
                                      >     # "Reset" the 1Hz spill frequency so t0 is wrt the spill start.
                                      >     # This is to enable the use of the modules/methods "out-of-the-box
                                      >     # The 1 sec. space between spills will be accounted for in the
                                      >     # packet timestamps through the event_times array below
                                      >
                                      >     #print('t0_start =',tracks['t0_start'])
                                      >     #print('t_start =',tracks['t_start'])
                                      >     tracks['t0_start'] = tracks['t0_start']%1e6
                                      >     tracks['t0_end'] = tracks['t0_end']%1e6
                                      >     tracks['t0'] = tracks['t0']%1e6
                                      >     #print('t0_start =',tracks['t0_start'])
                                      >
    event_times = fee.gen_event_times(tot_evids.max()+1, 0)       |     #event_times = fee.gen_event_times(tot_evids.max()+1, 0)
                                      >     #event_times = [0.]*(tot_evids.max()+1)
                                      >     #event_times = cp.array(np.zeros(tot_evids.max()+1),dtype='f8')
                                      >     #event_times = cp.array(tracks['spillID']*1E6) # new event/spill e
                                      >     event_times = cp.arange(tracks[EVENT_SEPARATOR].max()+1) * 1e6
                                      >
                                      >     #print("event_times =",list(event_times))
                                      >     #print("event_times.dtype =",event_times.dtype)
                                      >     #print("len(event_times) =",len(event_times))
                                      >
                                      <
                                      >
                                              |  
                                      <
                                      >             synchronize()
            BPG = ceil(pixels_signals.shape[0] / TPB)             |             BPG = max(ceil(pixels_signals.shape[0] / TPB),1)
            pixel_thresholds = pixel_thresholds_lut[unique_pix.ravel()    |             pixel_thresholds = cp.full(unique_pix.shape[0],7000.,float
                                      <
                                      <
                                      |             last_time = 9

The event separator stuff should all work fine though, but I might complain about the changing capitalization scheme...

[krwood]

I completely agree with your organizational comments. I was planning on merging the two simulate_pixel* scripts in a future PR (although you include some good ideas I wasn't considering). The reason for merging this branch in now is mostly to get the other edits in so that other outstanding PRs can be tested and merged as well (e.g. one outstanding now will need the fixes to batching.py to validate). I suggest we either (i) remove simulate_pixels_spills.py from this PR and merge or (ii) merge everything as is and create an issue to note the implementation you outlined in the initial message here.

[peter-madigan]

Ah ok. Either is fine by me.

[krwood]

@drinkingkazu would you mind giving this PR a look over and letting me know if it looks like it will work for your purposes? Your outstanding PR will need to be revised if you're happy to merge this branch in. Instead of making the EVENT_SEPARATOR argument configurable from the command line, this PR reads it in from a configuration file.