search

akash-network / awesome-akash

Awesome List of Akash Deployment Examples
298 stars 217 forks source link

add vllm support #519

Closed rakataprime closed 1 month ago

rakataprime commented 2 months ago

adds vllm blog and deployment examples. vllm is a multiuser llm server suitable for llm services with lots of concurrent users . @anilmurty

anilmurty commented 2 months ago

Thanks @rakataprime - as discussed on discord:

  1. The model field wasn't getting populated in the UI - if that's expected we should make a note to the user and provide some instructions on how to get that working
  2. In general, it'll be good to add some examples of both (UI and CLI) for how to use the app once it is deployed
Dimokus88 commented 2 months ago

Hey @rakataprime ! Please, add info in root README.md about this template.

rakataprime commented 2 months ago

@Dimokus88 @anilmurty This should be good to go. I have made changes to SDL and added a troubleshooting guide. Also someone else created and merged a lower quality PR for vllm a few days after after this pr was opened #521. It might be better to merge that example into a single folder rather than have two separate ones.

rakataprime commented 1 month ago

@anilmurty can you merge this in now ?