Example with VLLM

[vjagannath786]

It would have been nice if there were some inference examples like vLLM. So that we know how exactly model inferencing is working.

[AnthonyDipilato]

Inference is the same in vllm as any other model, they have examples in their documentation https://docs.vllm.ai/en/latest/getting_started/quickstart.html

[danielhanchen]

Yep vLLM has some cool docs - would adding it into an Unsloth notebook be helpful?

[vjagannath786]

Yep vLLM has some cool docs - would adding it into an Unsloth notebook be helpful?

Yes, it would be helpful. Actually I was trying to solve classification tasks which would churn out logits and was experimenting with this notebook https://github.com/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb which wasn't working and I had to do a fast inferencing vLLM. So that is where I am stuck.

Would be glad if some examples are added for vLLM and classification tasks.

[danielhanchen]

Ok will try add some vLLM examples!