search

TabbyML / tabby

Self-hosted AI coding assistant
https://tabbyml.com
Other
21.82k stars 995 forks source link

Vulkan Backend Support for improved device compatibility #124

Closed FruityWelsh closed 6 months ago

FruityWelsh commented 1 year ago

Please describe the feature you want Support for the pytorch vulkan backend so that older nvidia gpus, as well as intell, amd, and some phone gpus can be supported. https://pytorch.org/tutorials/prototype/vulkan_workflow.html

Additional context Personally ran into difficulties testing this project, because my laptop is too old to support Nvidia, and my cloud accounts aren't authorized to deploy GPU compute. I imagine I am not the only one limited on working on this project by these kinds of lim facs.

Please reply with a 👍 if you want this feature.

wsxiaoys commented 1 year ago

related：https://github.com/webonnx/wonnx/issues/169

wsxiaoys commented 9 months ago

The recent merges of two Vulkan-based implementations in llama.cpp are as follows:

  1. The first merge: https://github.com/ggerganov/llama.cpp/commit/2307523d322af762ae06648b29ec5a9eb1c73032
  2. The second merge: https://github.com/ggerganov/llama.cpp/commit/fbf1ddec69f7001cc707de17fa74d7200813bbac

We are quite interested in integrating one of these implementations into Tabby. For example, by adding a vulkan device option, similar to the metal option, we can make Tabby compatible with certain GPUs, such as the Intel Arc.

wsxiaoys commented 8 months ago

Note this is done in #1588 and a vulkan binary has been added in https://github.com/TabbyML/tabby/actions/runs/8179533225, consider give it a try!

cc @boxbeam