Multi trials for btb eval

HDI-Project / ATM

Auto Tune Models - A multi-tenant, multi-data system for automated machine learning (model selection and tuning).

https://hdi-project.github.io/ATM/

MIT License

527 stars 141 forks source link

Multi trials for btb eval #69

Closed lauragustafson closed 6 years ago

lauragustafson commented 6 years ago

Addresses #64. evaluate_btb.py now runs 10 independent trials using the specified tuner/selector/dataset combination. It compares the mean, minimum, and standard deviation of the AUC over the 10 trials to the baseline (calculated also from 10 independent trials). Test output: multi_trial_results If run ids are passed in, it compares the baseline to each specified run. Test output for specified run ids: multi_trials_existing_dataruns

bcyphers commented 6 years ago

Is this ready to go? if so I'll merge it.

lauragustafson commented 6 years ago

Yup.