Closed RobertTLange closed 3 years ago
It would also be nice to not only have an exclude_nodes
option in experiment .yaml
but to also restrict overall usage of each node on the cluster. E.g. say you want to leave 20% on each node (or specific ones) free. Denis may require a couple cores on cognition13
for GPU development. Maybe there is a grid engine command using CPU_LOAD
?!
This has been partially (but sufficiently 😄) addressed with issue #8 (asynchronous scheduling).
Currently it is not directly obvious how many jobs are being launched (e.g. at each iteration of a grid batch evaluation) since we are always launching all random seeds for a single experiment config at once. So effectively, we are always getting
num_evals_per_iter
*num_iter_per_batch
new experiment launches at the beginning of an episode.Ideas: Ideally this should be solved by setting a
total_number_running_jobs
at every given point in time. The toolbox then automatically takes care of how much is scheduled at any given point in time.This should potentially be addressed together with issue #8.