Closed ltm920716 closed 2 months ago
Hey @ltm920716, thanks for the report!
I believe this was fixed by the following PR https://github.com/huggingface/text-generation-inference/pull/1986 from @Narsil.
You'll now get this instead:
WARN text_generation_launcher: Could not import Flash Attention enabled models: `USE_FLASH_ATTENTION` is false.
I believe this hasn't been released in a version yet, but you can use the docker image that was released with the commit of the PR above:
ghcr.io/huggingface/text-generation-inference:sha-06edde9
The resulting command would be:
sudo nerdctl run -it --gpus '"device=1"' -e USE_FLASH_ATTENTION=False -p 8080:80 -v /data/models:/data ghcr.io/huggingface/text-generation-inference:sha-06edde9 --model-id /data/Meta-Llama-3-8B-Instruct
thanks a lot, I will have a try!
System Info
Linux k8s-node2 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0
NVIDIA-SMI 535.171.04 Driver Version: 535.171.04
P40
Information
Tasks
Reproduction
log ====>
ImportError:
USE_FLASH_ATTENTION
is false.Expected behavior
run normally on P40
help please, thanks!