search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
29k stars 4.32k forks source link

classification with BartForSequenceClassification? #1187

Closed silvacarl2 closed 7 months ago

silvacarl2 commented 1 year ago

another huge problem requiring speed is BartForSequenceClassification ala "facebook/bart-large-mnli"

viktor-ferenczi commented 1 year ago

What would you suggest exactly? Could you please detail this?

silvacarl2 commented 1 year ago

without having to create a fine tuned classification model, "facebook/bart-large-mnli" does a very good job. as part of a vllm "toolkit", if it supports mistralai/Mistral-7B-Instruct-v0.1, whisper, and facebook/bart-large-mnli, all running blazing fast, ANYTHING can be built out of it. Literally anything.

if through testing mistralai/Mistral-7B-Instruct-v0.1 can do as well or better than facebook/bart-large-mnli at categorization, then it would be uneccsary but we have not proved yet if that is the case or not.

right now mistralai/Mistral-7B-Instruct-v0.1 appears to be the best 7B model we have seen, so we are going to test it witlh vllm.