optuna / kurobako

A black-box optimization benchmark tool
MIT License
83 stars 11 forks source link

Add batch-evaluate mode #47

Closed contramundum53 closed 3 months ago

contramundum53 commented 2 years ago

Kurobako has evaluate subcommand that evaluates a problem at a given set of params. However, when we want to repeatedly evaluate different sets of params and the problem needs to load a large dataset, this is inefficient because the problem reloads the dataset every time. We add batch-evaluate subcommand that uses stdio to repeatedly evaluate sets of params without reloading the process.

coveralls commented 2 years ago

Pull Request Test Coverage Report for Build 2961163548

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details


Changes Missing Coverage Covered Lines Changed/Added Lines %
src/main.rs 0 4 0.0%
src/batch_eval.rs 0 37 0.0%
<!-- Total: 0 41 0.0% -->
Files with Coverage Reduction New Missed Lines %
../../../.cargo/registry/src/github.com-1ecc6299db9ec823/trackable-0.2.23/src/macros.rs 5 0.0%
../../../.cargo/registry/src/github.com-1ecc6299db9ec823/ndarray-0.12.1/src/error.rs 6 0.0%
kurobako_core/src/registry.rs 8 0.0%
kurobako_core/src/trial.rs 10 0.0%
kurobako_problems/src/sigopt.rs 40 0.0%
src/plot/slice.rs 45 0.0%
src/plot/pareto_front.rs 46 0.0%
src/plot/curve.rs 98 0.0%
<!-- Total: 258 -->
Totals Coverage Status
Change from base Build 2668460033: -0.02%
Covered Lines: 1092
Relevant Lines: 11510

💛 - Coveralls
sile commented 2 years ago

FYI: I just created https://github.com/optuna/kurobako/pull/48 to fix CI (lint) failure.

HideakiImamura commented 3 months ago

Let me merge this PR with my one approval since we need this feature.