search

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with KoboldAI's UI
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.35k stars 312 forks source link

Cannot run --flashattention on arch without crashing #888

Closed scrubborn closed 4 weeks ago

scrubborn commented 4 weeks ago 
ggml_vulkan: Error: Missing op: FLASH_ATTN_EXT
GGML_ASSERT: ggml-vulkan.cpp:5615: false
gdb: /tmp/_MEI1NGty2/libstdc++.so.6: version `CXXABI_1.3.15' not found (required by /usr/lib/libicui18n.so.75)
fish: Job 1, './koboldcpp-linux-x64-nocuda --…' terminated by signal SIGABRT (Abort)

I do appear to have the proper libstdc installed on my machine but it is not located in temp, that whole "_MEI1NGty2" folder is no where to be found. I tried "export LD_LIBRARY_PATH=/usr/local/lib64/:$LD_LIBRARY_PATH " to get it to look in the proper place and it didn't work. I can't find any info online as to what to do here.

Edit: my launch command is this -
./koboldcpp-linux-x64-nocuda --usevulkan --gpulayers 33 --contextsize 4096 --blasbatchsize 2048 --flashattention '/home/scrubby/SSD/text-generation-webui/models/mythomist-7b.Q3_K_M.gguf'

LostRuins commented 4 weeks ago

The issue is because you picked vulkan, flash attn is not implemented there

scrubborn commented 4 weeks ago

Gotcha, I couldn't find any documentation on that. I'll close the issue.