search

mle-infrastructure / mle-toolbox

Lightweight Tool to Manage Distributed ML Experiments 🛠
https://mle-infrastructure.github.io/mle_toolbox/toolbox/
MIT License
2 stars 0 forks source link

General Processing Job - Pre & Post #75

Closed RobertTLange closed 3 years ago

RobertTLange commented 3 years ago

Currently we do have the option to launch a post-processing job, e.g. to create figures after a grid search was completed:

https://github.com/RobertTLange/mle-toolbox/blob/4e0f78f2ccfda898e6083f17cfca9f1b1b97be05/mle_toolbox/experiment/spawn_jobs.py#L35-L55

But we don't have a way to launch a pre-processing job, e.g. to do some data wrangling that is then later on used in a search experiment. So I would like to generalize this setup. To simply a spawn_processing_job or get rid of processing in the naming.

Towards general sequential experiment composition! 🚀 Also get rid of random figure directory storing in protocol.

RobertTLange commented 3 years ago

Addressed in 6dcfc122916ce1cb8e785bd4e55f2ee20cd617ba and 2a25b6cfa1fa3609eedfcff79512575e093d5a90. Checkout the example for the torch MNIST-CNN.