search

henk717 / KoboldAI

KoboldAI is generative AI software optimized for fictional use, but capable of much more!
http://koboldai.com
GNU Affero General Public License v3.0
352 stars 130 forks source link

Exllama2 GPU Split #505

Closed one-some closed 3 months ago

one-some commented 5 months ago

Implements a wrapper class (LayerSplitExLlamaV2) to hjijack ExLlamaV2's set_device_map to deal in number of hidden layers instead of doing fancy storage calculations

Tested on 13B_Ouroboros_GPTQ as I don't have many compatible models on hand