Support older GPUs without shaderFloat16

nomic-ai / gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

https://nomic.ai/gpt4all

MIT License

70.46k stars 7.69k forks source link

Support older GPUs without shaderFloat16 #1119

Open sukualam opened 1 year ago

sukualam commented 1 year ago

System Info

it seems running on cpu only while my gpu (amd rx560) is actually supported latest metal and can run stable diffusion on macos. why force metal is for arm only? why it cant intel with amd gpu ?

Information

[ ] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[X] backend
[ ] bindings
[ ] python-bindings
[ ] chat-ui
[X] models
[ ] circleci
[ ] docker
[ ] api

Reproduction

checked force metal

Expected behavior

utilise my amd gpu instead cpu

niansa commented 1 year ago

Hmm, does it support Vulkan?

cebtenzzre commented 11 months ago

You can check if your GPU is currently supported by going to https://vulkan.gpuinfo.org/, finding your device, and searching for shaderFloat16 under "Features > Core 1.2". GPT4All currently requires this.

vap0rtranz commented 11 months ago

Thanks for that link.

Just to clarify. GPT4All after v2.5 supports AMD if the GPU supports Vulkcan >v2.1? and specifically shaderFloat16?

I read some other issues about GPU as well and it sounds like a few more requirements are: having enough contiguous VRAM, and using only q4 or q1 models.

Did I get that about right?

cebtenzzre commented 11 months ago

Did I get that about right?

It's Vulkan 1.2, and either q4_0 or q4_1 models. Otherwise, yeah, that's the current state of affairs.

vap0rtranz commented 10 months ago

Thanks. I meant Vulkan v1.2 but dyslexia sucks.