Support multi-objective optimization

anencore94 commented 3 years ago

/kind feature

Describe the solution you'd like

katib currently use only single-objective optimization, (even though katib could display additional metrics, the optimizing process focused on single objective function)
However, as you might know well, optuna support multi-objective optimization
- which means, for example, we can optimize the hyperparameters to maximize accuracy and minimize time-cost simultaneously.
In such case, I know we can define a new objective function which combines two objectives, such as maximize accuracy / time-cost or something else more complicated.
However, I wonder is there any plan to support such feature natively in katib.

andreyvelich commented 3 years ago

Thank you for creating this @anencore94. I think that feature will require some API modification since type, goal and objectiveMetricName can be unique: https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/common/v1beta1/common_types.go#L94-L103.

@anencore94 @gaocegege @johnugeorge Any ideas how we can implement this ?

@c-bata Does Goptuna support Multi-object optimization ?

c-bata commented 3 years ago

@c-bata Does Goptuna support Multi-object optimization ?

No. I'm not planning to implement multi-objective support in Goptuna because it requires some breaking changes.

As @anencore94 said, I think Optuna is a good choice if Katib will support multi-objective optimizations. Optuna supports a lot of multi-objective optimization algorithms like NSGA-II, multi-objective TPE, and the algorithms provided by BoTorch. And it is not difficult to implement Optuna based suggestion service since its internal design is almost similar with Goptuna.

g-votte commented 3 years ago

Hi, I'm one of the maintainers of Optuna. Thank you for referring to Optuna as a possible backend of the multi-objective API.

As @c-bata mentions, Optuna supports several algorithms for multi-objective optimization, with almost the same interface as single objective optimization.

I think we can go forward by (1) implementing the Optuna suggestion service, and (2) fixing the interface of Katib's multi-objective functionality.

For (1), I created the PR #1613 to add the suggestion service. I appreciate your taking a look and providing any feedback.

anencore94 commented 2 years ago

Can we close this issue by #1613 ?

andreyvelich commented 2 years ago

Can we close this issue by #1613 ?

This issue still is progress. If we want to support multi-objective optimisation, we have to modify our Katib Experiment APIs.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

andreyvelich commented 1 year ago

/lifecycle frozen /help

google-oss-prow[bot] commented 1 year ago

@andreyvelich: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubeflow/katib/issues/1549): >/lifecycle frozen >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

Padarn commented 1 year ago

I saw this issue mentioned on the 2023 roadmap, but it looks like the PR above resolves it? There is an example in the PR https://github.com/kubeflow/katib/pull/1613/files#diff-1057ae2ec900a6062a9c0c6e12e9845a4ae2c29f5cf4ae1bf9265ab6b227155c

andreyvelich commented 1 year ago

I saw this issue mentioned on the 2023 roadmap, but it looks like the PR above resolves it? There is an example in the PR https://github.com/kubeflow/katib/pull/1613/files#diff-1057ae2ec900a6062a9c0c6e12e9845a4ae2c29f5cf4ae1bf9265ab6b227155c

@Padarn We haven't implemented multi-objective optimization using Optuna yet. We are looking for contributors to support this feature. If you are interested, I am happy to discuss this in one of our upcoming Community Calls.

kubeflow / katib

Support multi-objective optimization #1549