Each sub-package has its own set of (unit) tests and the toolbox test suite will mainly consist of integration tests. This way everyone can use every sub-package individually as well as from within the integrating mle-toolbox. So what does this mean in terms of restructuring?
Let's move all of the functionality in the job sub-directory into mle-scheduler and import the main job and queue as MLEJob and MLEQueue. Get of all mle_config shenanigans in the sub-package.
The mle-toolbox will ultimately implement all of the following:
All of the mle subcommands.
Different experiment types (multi-config, search, etc.) and their scheduling.
Report generation and figure generation.
MLELauncher for launching experiments from a notebook.
Auto-setup (via mle_config) and glueing together of sub-packages.
Local execution of remote jobs (Copying of files, etc.).
More experimental utilities (hypothesis testing, PBT)
Everything should be easily extendable for any additional cloud resource (Azure, AWS).
I believe that it is probably the most elegant if the overall
mle-infrastructureconsists of 5 packages:mle-logging: Logging of individual experiment runs.mle-hyperopt: Hyperparameter search strategies.mle-monitor: Experiment protocol & resource utilisation monitoring/dashboardmle-scheduler: Functionalities for launching jobs and monitoring their statusmle-toolbox: Glues everything together & provides post-processing analysis toolsEach sub-package has its own set of (unit) tests and the toolbox test suite will mainly consist of integration tests. This way everyone can use every sub-package individually as well as from within the integrating
mle-toolbox. So what does this mean in terms of restructuring?Let's move all of the functionality in the
jobsub-directory intomle-schedulerand import the main job and queue asMLEJobandMLEQueue. Get of allmle_configshenanigans in the sub-package.The
mle-toolboxwill ultimately implement all of the following:mlesubcommands.MLELauncherfor launching experiments from a notebook.mle_config) and glueing together of sub-packages.Everything should be easily extendable for any additional cloud resource (Azure, AWS).