Open vjagannath786 opened 1 month ago
Inference is the same in vllm as any other model, they have examples in their documentation https://docs.vllm.ai/en/latest/getting_started/quickstart.html
Yep vLLM has some cool docs - would adding it into an Unsloth notebook be helpful?
Yep vLLM has some cool docs - would adding it into an Unsloth notebook be helpful?
Yes, it would be helpful. Actually I was trying to solve classification tasks which would churn out logits and was experimenting with this notebook https://github.com/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb which wasn't working and I had to do a fast inferencing vLLM. So that is where I am stuck.
Would be glad if some examples are added for vLLM and classification tasks.
Ok will try add some vLLM examples!
It would have been nice if there were some inference examples like vLLM. So that we know how exactly model inferencing is working.