Under the current workflow, after a training run is queued on HPC resources, train_config.yaml must remain untouched - at least until the run starts - because Snakemake will read configuration info from that file upon starting. This means that a new run with changes to the configuration file musn't be started until the old run reaches the front of the queue and begins. That's a bottleneck for our workflow.
A more robust solution would be to use separate config files for each run.
Under the current workflow, after a training run is queued on HPC resources,
train_config.yamlmust remain untouched - at least until the run starts - because Snakemake will read configuration info from that file upon starting. This means that a new run with changes to the configuration file musn't be started until the old run reaches the front of the queue and begins. That's a bottleneck for our workflow.A more robust solution would be to use separate config files for each run.