search

kubernetes-sigs / llm-instance-gateway

LLM Instance gateway implementation.
Apache License 2.0
84 stars 15 forks source link

Switch to upstream vLLM #22

Open liu-cong opened 1 month ago

liu-cong commented 1 month ago

We used a forked vLLM for POC for additional metrics not available in vLLM. With https://github.com/vllm-project/vllm/pull/9477/files, we have the required LoRA metrics in vLLM. This issue involves the following tasks:

liu-cong commented 1 month ago

/assign @coolkp

k8s-ci-robot commented 1 month ago

@liu-cong: GitHub didn't allow me to assign the following users: coolkp.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to [this](https://github.com/kubernetes-sigs/llm-instance-gateway/issues/22#issuecomment-2427122862): >/assign @coolkp Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
coolkp commented 1 month ago

/assign @coolkp

k8s-ci-robot commented 1 month ago

@coolkp: GitHub didn't allow me to assign the following users: bot, assign, to, me.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to [this](https://github.com/kubernetes-sigs/llm-instance-gateway/issues/22#issuecomment-2428078104): >/assign @coolkp bot assign to me Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.