Closed infophysics closed 9 months ago
Set up arrakis and dataset as modules, need to refine the dataset module to be general, so blip_dataset needs a lot of refactoring. Arrakis part doesn't do singles correct yet, its kind of cludgey at the moment, but will fix in the future. Need to still work on having multiple configs being imported or run sequentially.
Cleaned up the dataset a bit. Need to add other example datasets to test its generality. Needs more cleaning up of the Blip part. Arrakis singles will be solved by updating Arrakis to be able to relabel things according to a fcl specification. More config import options to be added as well. Arrakis needs to have a section for downloading datasets from OSF/GitLFS so that this can be done in batch jobs on Perlmutter.
Other dataset types still need to be added. Add a dataset template that people can use.
The arrakis and blip dataset steps should be done as actual module steps, which append their respective outputs to the meta dictionary that is then fed into the other modules. Blip dataset needs a bunch of cleaning up and refactoring, arrakis needs to have the singles generation added to it. Configs also need to be set up so that separate config files can be run for different modules. Something like: