search

mle-infrastructure / mle-toolbox

Lightweight Tool to Manage Distributed ML Experiments 🛠
https://mle-infrastructure.github.io/mle_toolbox/toolbox/
MIT License
3 stars 1 forks source link

Aggregation of logs over multiple experiments #84

Closed RobertTLange closed 3 years ago

RobertTLange commented 3 years ago

Make it easy to aggregate logs over multiple different grid experiments (e.g. when running benchmarking experiments) over different models/optimizers/algorithms - form of composable meta-meta log?!

Given a list of experiment directories and a list of string descriptions, aggregate different meta and hyper log files into single files.

For meta-log add a level of hierarchy:

meta_log
--> experiment_id
       --> config_id
              --> seed_id

For hyper-log nest different pd dataframes:

hyper_hyper_df = pd.DataFrame({'idx':[1,2,3], 'dfs':[hyper_df1, hyper_df2, hyper_df3]})

which can then be indexed via

hyper_hyper_df['dfs'].iloc[0]

P.S.: The meta-log part could also happen in mle-logging. Unsure about this.

RobertTLange commented 3 years ago

Addressed in 6e5cc5897da1cb8bd21c55749d0197f66d07bc8e by combine_experiments