Neutron-Calibration-in-DUNE / Blip

Blip ML/TDA package for LArTPCs.
0 stars 0 forks source link

Move arrakis and blip_dataset prep into separate modules which are run automatically during the ML module step, but are moved out of the wrapper. Set up separate config usage. #44

Closed infophysics closed 9 months ago

infophysics commented 11 months ago

The arrakis and blip dataset steps should be done as actual module steps, which append their respective outputs to the meta dictionary that is then fed into the other modules. Blip dataset needs a bunch of cleaning up and refactoring, arrakis needs to have the singles generation added to it. Configs also need to be set up so that separate config files can be run for different modules. Something like:

module:
  module_name: "optimize_blip_graph"
  module_type: ['arrakis', 'data_prep', 'ml']
  module_mode: ['larsoft_singles', 'view_cluster', 'contrastive_hyper_parameter_scan']
  module_configs: ['<larsoft_config>.yaml', '<data_prep_config>.yaml', '<optimize_config>.yaml']
  ...
infophysics commented 11 months ago

Set up arrakis and dataset as modules, need to refine the dataset module to be general, so blip_dataset needs a lot of refactoring. Arrakis part doesn't do singles correct yet, its kind of cludgey at the moment, but will fix in the future. Need to still work on having multiple configs being imported or run sequentially.

infophysics commented 11 months ago

Cleaned up the dataset a bit. Need to add other example datasets to test its generality. Needs more cleaning up of the Blip part. Arrakis singles will be solved by updating Arrakis to be able to relabel things according to a fcl specification. More config import options to be added as well. Arrakis needs to have a section for downloading datasets from OSF/GitLFS so that this can be done in batch jobs on Perlmutter.

infophysics commented 10 months ago

Other dataset types still need to be added. Add a dataset template that people can use.