search

theroyallab / tabbyAPI

An OAI compatible exllamav2 API that's both lightweight and fast
GNU Affero General Public License v3.0
503 stars 67 forks source link

Model: Fix draft model loading #175

Closed veden closed 1 month ago

veden commented 1 month ago

Is your pull request related to a problem? Please describe. The problem is that when a draft model is added and the cache class is attempting to load the self.model object which is None and .config cannot be found because the model that is loaded at the time of draft cache creation is self.draft_model

Why should this feature be added? Who doesn't enjoy extra tokens per second.

veden commented 1 month ago

main branch has separate fix, closing as this is no longer needed.