feat: Add blueprint for using RayServe with vLLM on gpus - Githubissues

awslabs / data-on-eks

DoEKS is a tool to build, deploy and scale Data & ML Platforms on Amazon EKS

https://awslabs.github.io/data-on-eks/

Apache License 2.0

550 stars 180 forks source link

feat: Add blueprint for using RayServe with vLLM on gpus #563

Open ratnopamc opened 4 days ago

ratnopamc commented 4 days ago

What does this PR do?

Adds capability for LLM inference using Rayserve with vLLM backend
Adds Serving script with RayServe, vLLM and FastAPI
Adds client python app
Adds website doc for end-to-end deployment and testing

🛑 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted. Consult the CONTRIBUTING guide for submitting pull-requests.

Motivation

Closes #547

More

[x] Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
[x] Mandatory for new blueprints. Yes, I have added a example to support my blueprint PR
[x] Mandatory for new blueprints. Yes, I have updated the website/docs or website/blog section for this feature
[x] Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

For Moderators

[ ] E2E Test successfully complete before merge?

Additional Notes