feat(llm): Determine the best LLM deployment config automatically

[gaocegege]

What you would like to be added?

Inspired by this research paper Vidur: A Large-Scale Simulation Framework For LLM Inference

Optimizing the deployment of Large language models (LLMs) is expensive today since it requires experimentally running an application workload against an LLM implementation while exploring large configuration space formed by system knobs such as parallelization strategies, batching techniques, and scheduling policies.

we present Vidur-Search, a configuration search tool that helps optimize LLM deployment. Vidur-Search uses Vidur to automatically identify the most cost-effective deployment configuration that meets application performance constraints. For example, Vidur-Search finds the best deployment configuration for LLaMA2-70B in one hour on a CPU machine, in contrast to a deployment-based exploration which would require 42K GPU hours – costing 218K dollars.

Why is this needed?

Not sure if it is in the scope of katib, but glad to raise an issue here.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[Electronic-Waste]

I guess it may belong to the scope of KServe, since Katib focuses on the hyperparameters tuning of models :)

[gaocegege]

It's more like a tuning job. You can consider tuning the deployment configs. (e.g. distributed strategy)

[andreyvelich]

Thank you for creating this @gaocegege!

Yes, I think optimization of LLM Deployment makes sense since Katib is able to perform any optimization task (not even ML) and orchestrate any resources as Trials.

It would be nice to get someone from the Kubeflow community who can explore the Vidur aspects and see how Katib can be useful.

/help /area llm /remove-label lifecycle/needs-triage

[google-oss-prow[bot]]

@andreyvelich: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubeflow/katib/issues/2396): >Thank you for creating this @gaocegege! > >Yes, I think optimization of LLM Deployment makes sense since Katib is able to perform any optimization task (not even ML) and orchestrate any resources as Trials. > >It would be nice to get someone from the Kubeflow community who can explore the Vidur aspects and see how Katib can be useful. > >/help >/area llm >/remove-label lifecycle/needs-triage > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[google-oss-prow[bot]]

@andreyvelich: The label(s) /remove-label lifecycle/needs-triage cannot be applied. These labels are supported: tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to [this](https://github.com/kubeflow/katib/issues/2396#issuecomment-2269046310): >Thank you for creating this @gaocegege! > >Yes, I think optimization of LLM Deployment makes sense since Katib is able to perform any optimization task (not even ML) and orchestrate any resources as Trials. > >It would be nice to get someone from the Kubeflow community who can explore the Vidur aspects and see how Katib can be useful. > >/help >/area llm >/remove-label lifecycle/needs-triage > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[gjyotin305]

@gaocegege @andreyvelich I would love to look into this,can I work on this ?

/assign