search

theroyallab / tabbyAPI

An OAI compatible exllamav2 API that's both lightweight and fast
GNU Affero General Public License v3.0
591 stars 75 forks source link

[REQUEST] KoboldAI Horde support #143

Closed bdashore3 closed 3 months ago

bdashore3 commented 4 months ago

Why should this feature be added? Since tabby has batching, it should be able to safely act as a KoboldAI horde worker. However, adding support requires understanding the Kobold API and adding a method to configure which API to use (OAI/Kobold). In addition, common routes are required for admin tasks such as loading and unloading.

Additional context Help wanted for Kobold API documentation and understanding the API as a whole. Also need help understanding how horde works.

Kobold's Wiki

bdashore3 commented 3 months ago

Added in b7cb6f0b91c9299ca447107cac82f5dc65ca4d59. Thanks @AlpinDale for helping with the typings.