This is something I have been working on for a few years, and it has reached a good level. I'll adapt it and contribute it here. Open to your thoughts.
The basic idea is:
You have your arg parser with your default arguments, their keys and values.
You want a way to select a subset and search over some space you want to define.
You use a script where all you need to define is a nice tuple of the variable names you want to search overs as well as the values you want to search over. The script finds all possible combinations, and generates a json named automatically using the default experiment name + the arguments that vary.
You then use a script that picks up any templates in a folder with 'experiment_script_templates', e.g. one for CDT cluster, one for GPU boxes like edwards etc and generates experiment scripts by combining them with all the available experiment json files from 3.
With a measly 2 clicks you get every single combo of the stuff you want to search over in a script format ready to run on the machine of your choice.
AntreasAntoniou
commented
3 years ago
This is something I have been working on for a few years, and it has reached a good level. I'll adapt it and contribute it here. Open to your thoughts.
The basic idea is:
With a measly 2 clicks you get every single combo of the stuff you want to search over in a script format ready to run on the machine of your choice.