Open n-gao opened 2 years ago
In #79 Daniel mentioned that this should allow overriding Slurm parameters when starting an experiment, which is not addressed by this design.
However, I don't think it's ideal to have start
do side-effects like changing the database. I'd rather propose a new command (update
?) for updating config values. This would combine the filtering syntax of start
with the overriding syntax of this and #79:
seml <collection> update -b 7 -sb mem=25GB -o dataset=imagenet
I'd suggest to add the overriding options to both add
and update
then.
Having an update command raises the potential issue that an overwrite parameter could cause two identical configurations. We would then have to check for duplicates again and potentially remove these. Still, having an override option at add time seems beneficial to me.
I'd suggest breaking this up into two issues, as these are more or less orthogonal:
update
command.Yes, good point! Let's keep the PRs small and manageable.
update could also break our assumption that every batch has consistent Slurm parameters. Creating batch arrays and sharing GPUs both rely on this. As a first step we could only allow changing Slurm parameters of full batches (and document the reasoning behind these restrictions).
79 follow-up.
At the moment, one must specify
sbatch
options in theyaml
file. However, as discussed in #71, it may be beneficial to override these settings via the CLI. The suggested syntax would be:The new syntax should be adapted to the
-sb
option inseml jupyter
as well. For parsing the key, values we can reuse the key value parser introduced in #79.