Issue listing useful docker containers for large scale maintenance and experiment tasks:
running check_estimator on all estimators or a subset thereof (e.g., specified by scitype). Runtime performance and memory benchmark, saves results by estimator, test and/or test-fixture combination as returned by check_estimator. With multiple reruns.
running evaluate or benchmark experiments on a selected subset of estimators, producing a typical research paper benchmarking table. We should start with the forecasting benchmark class, as other tasks will move to a similar design.
FYI @benHeid, @hazrulakmal due to benchmarking; @astrogilda due to research; @shlok191 due to recent discussion; @YelenaYY due to project, and @SaiRevanth25 due to https://github.com/sktime/sktime/issues/6555
Issue listing useful docker containers for large scale maintenance and experiment tasks:
check_estimator
on all estimators or a subset thereof (e.g., specified by scitype). Runtime performance and memory benchmark, saves results by estimator, test and/or test-fixture combination as returned bycheck_estimator
. With multiple reruns.evaluate
or benchmark experiments on a selected subset of estimators, producing a typical research paper benchmarking table. We should start with the forecasting benchmark class, as other tasks will move to a similar design.FYI @benHeid, @hazrulakmal due to benchmarking; @astrogilda due to research; @shlok191 due to recent discussion; @YelenaYY due to project, and @SaiRevanth25 due to https://github.com/sktime/sktime/issues/6555