Closed Zijie-Tian closed 1 month ago
Hi @Zijie-Tian , thanks for your interest. T-MAC needs to use clang compiler for cpu,while cuda needs nvcc compiler, so it is recommended to set "LLAMA_TMAC=OFF" when compiling cuda.
Hi, @caoshijie0501 Sorry for late response, I am very interested in your work. Using your method, I was able to compile llama-bench and main with CUDA now.
However, after reading the paper, I have the following questions for you:
Thank you !!!
Hi, @caoshijie0501 Sorry for late response, I am very interested in your work. Using your method, I was able to compile llama-bench and main with CUDA now.
However, after reading the paper, I have the following questions for you:
- Regarding extremely low-bit precision, I am curious about how you setup your baseline. For example, I saw the results for llama (W2) in your profile_data.md. How is these special precision executed on the CPU/GPU side?
- How can Hugging Face models be exported to these special precisions?
Thank you !!!
As stated in README.md,
We evaluate BitNet-3B and Llama-2-7B (W2) with T-MAC 2-bit and llama.cpp Q2_K, and evaluate Llama-2-7B (W4) with T-MAC 4-bit and llama.cpp Q4_0.
You can refer to llama.cpp for how to obtain the quantized q2k/q40 models or convert from huggingace models, and how to execute the models with llama.cpp.
Currently we have provided support for BitNet/EfficientQAT/GPTQ. You can first start from the pretrained quantized models provided by BitNet/EfficientQAT following the doc. If you want to evaluate a random huggingface unquantized models with t-mac, you can refer to GPTQModel or BitDistiller on how to quantize a model and follow our readme to evaluate it.
@Zijie-Tian As the issue described by the title is already solved, I will close it. For more questions related to this repo, please open a new issue. And for questions more related to the paper rather than the code, please open a discussion.
I am trying to reproduce your experimental results, so I am testing with the llama.cpp you provided. However, when I enable CUDA compute capability using the following command, the following error occurs.
Output as following :
Could you please let me know how to modify llama.cpp and compile it with CUDA, or provide instructions on how to achieve your CUDA Baseline? If possible, it would be great if you could provide a complete script for Power experiment.