Flexibility to submit minimal `bash`-based experiments

mle-infrastructure / mle-toolbox

Lightweight Tool to Manage Distributed ML Experiments 🛠

https://mle-infrastructure.github.io/mle_toolbox/toolbox/

MIT License

3 stars 1 forks source link

Flexibility to submit minimal `bash`-based experiments #47

Closed RobertTLange closed 3 years ago

RobertTLange commented 3 years ago

Right now everything in the "inner loop" of the experiment relies on Python. It should be easy to take any Python-based code base and with minimal changes (mostly logging and how configs/parameters are handled) adapt it to be used with the mle_toolbox. But sometimes you may not work with Python. It would be great to also provide a minimal version of passing inputs via a shell script.

[Rob Note: This will be useful for the next set of experiments extending Marc's LTH code base.]

RobertTLange commented 3 years ago

Some thoughts:

There is basically only one place (for slurm/sge/local) where state that we want to call a python script. E.g.:

https://github.com/RobertTLange/mle-toolbox/blob/c3b03ce0454d04e8ccd2e52487452b8837dd7893/mle_toolbox/experiment/sge_job_management.py#L116

We could add the programming language as an argument to the experiment config .yaml and keep python as the default.
Alternatively, we could another primitive job submission function like spawn_post_processing_job. Although I am not 100% happy with that in the first place.

RobertTLange commented 3 years ago

Addressed in PR #51. Note that the user does not have to provide "backend" language. This is automatically handled via the file extension provided in the experiment.yaml file. Benefit: Can later also add C++ or other languages.

Note to self: Closing this for now, but might still changes once I fully get Marc's lottery ticket pipeline going.