To get us started, I have created a folder /pipeline to store scripts for the individual steps of the ML pipeline. I have created a second directory /slurm which will store the scripts for generating the slurm jobs (using job arrays, dependencies, etc...). The directory /src will contain the actual package code which, at this point, I intend to only be used for defining the auxiliary functions we will need for the pipeline. This will help us keep the dependencies minimal.
For applying the pipelines to specific datasets, I have created a /projects folder with subfolders for each individual dataset we want to analyze. Each of these can carry it's own Project.toml file with any extra dependencies for dataset-specific scripts.
To get us started, I have created a folder
/pipelineto store scripts for the individual steps of the ML pipeline. I have created a second directory/slurmwhich will store the scripts for generating the slurm jobs (using job arrays, dependencies, etc...). The directory/srcwill contain the actual package code which, at this point, I intend to only be used for defining the auxiliary functions we will need for the pipeline. This will help us keep the dependencies minimal.For applying the pipelines to specific datasets, I have created a
/projectsfolder with subfolders for each individual dataset we want to analyze. Each of these can carry it's ownProject.tomlfile with any extra dependencies for dataset-specific scripts.