[Examples] Add vLLM example

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[bikash119]

Curious to know how vllm is different from this issue?

[peterschmidt85]

Curious to know how vllm is different from this issue?

@bikash119 We'd like to have it as a separate example under the Deployment category (in addition to other categories such as Fine-tuning, Accelerators, etc.).

The example should feature most essential information around using vLLM with dstack.

Note, dstack allows to use vllm with both tasks and services. Should we show both, also explaining briefly the difference?

[bikash119]

Apologies for being a noob here. Under Deployment category means, to provide vllm as a serverless inference service on dstack?

[peterschmidt85]

Apologies for being a noob here. Under Deployment category means, to provide vllm as a serverless inference service on dstack?

@bikash119 Currently, there are three categories on https://dstack.ai/examples/: Fine-tuning, Accelerators, and LLMs. Let's add Deployment - can be the first in the list.

[bikash119]

Ok, so under Deployment , we should have 2 examples shown

• How to use vLLM for service
• How to use vLLM for task Does this sound ok?

[peterschmidt85]

Ok, so under Deployment , we should have 2 examples shown

• How to use vLLM for service
• How to use vLLM for task Does this sound ok?

Under Deployment, we are going to have one card "vLLM".

This will lead to https://dstack.ai/examples/deployment/vllm.

The sourcecode must be in examples/examples/deployment/vllm/README.md (it's copied to the docs when docs are built). See other cards on https://dstack.ai/examples/ to follow the structure and style.

Let me know if this helps

[bikash119]

Thank you @peterschmidt85 for being patient with my questions.

[bikash119]

@peterschmidt85 : I have made the required changes. Verifed by executing mkdocs serve

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.