Hi! While testing the model with vLLM https://github.com/vllm-project/vllm/issues/1913 I found out that the KV head count seemed strange, why is it 32 instead of the 16 like the base sheared llama?
Is it safe for me to just change that value to 16 and use the model that way?
Thanks for the great work! I've dreamed about just pruning LLMs for speculative decoding instead of training a separate model for a long time! :)
Hi! While testing the model with vLLM https://github.com/vllm-project/vllm/issues/1913 I found out that the KV head count seemed strange, why is it 32 instead of the 16 like the base sheared llama?
Is it safe for me to just change that value to 16 and use the model that way?
Thanks for the great work! I've dreamed about just pruning LLMs for speculative decoding instead of training a separate model for a long time! :)