Training settings are loaded via Snakemake from the config file 3_train/train_config.yaml. That means that running multiple training runs in parallel with different configs is problematic. Rather than use Snakemake and 3_train/train_config.yaml to load settings for training, this PR changes to storing configuration files in 3_train/in and loading them directly in Python, not through Snakemake as the current Snakefile's config. The format of the config file remains unchanged.
We could do the same for 2_process/process_config.yaml, but since I don't anticipate compiling training sets in parallel, I don't think it's worth the effort right now.
How to review this PR
This one's all in the snakefile! 3_train.smk, to be specific. It does complicate the snakefile a bit, so any suggestions about how to help clarify the process via comments or code tweaks would be appreciated.
Training settings are loaded via Snakemake from the config file
3_train/train_config.yaml. That means that running multiple training runs in parallel with different configs is problematic. Rather than use Snakemake and3_train/train_config.yamlto load settings for training, this PR changes to storing configuration files in3_train/inand loading them directly in Python, not through Snakemake as the current Snakefile's config. The format of the config file remains unchanged.We could do the same for
2_process/process_config.yaml, but since I don't anticipate compiling training sets in parallel, I don't think it's worth the effort right now.How to review this PR
This one's all in the snakefile!
3_train.smk, to be specific. It does complicate the snakefile a bit, so any suggestions about how to help clarify the process via comments or code tweaks would be appreciated.