awslabs / data-on-eks

DoEKS is a tool to build, deploy and scale Data & ML Platforms on Amazon EKS
https://awslabs.github.io/data-on-eks/
Apache License 2.0
617 stars 210 forks source link

Mistral 7B with vLLM, Ray Serve on Trn/Inf #650

Open askulkarni2 opened 2 weeks ago

askulkarni2 commented 2 weeks ago

Community Note

What is the outcome that you are trying to reach?

A pattern that demonstrates running the Mistral 7B model with the cheapest Neuron instances.

Describe the solution you would like

Similar to the llama3-8B-Instruct pattern but for Mistral instead