KVCache uses float16 by default.
Baichuan, Qwen and YaRNLlama previously defaulted to using FP32, but now default to using FP16 as well.
Usage:
C++ example:
example -t /data/llama-2-7b-hf/tokenizer.model -m /data/llama-2-7b-xft/ -d bf16 --kv_cache_dtype fp16
Python API:
model = xfastertransformer.AutoModel.from_pretrained(
args.model_path, dtype=args.dtype, kv_cache_dtype=args.kv_cache_dtype
)
KVCache uses float16 by default. Baichuan, Qwen and YaRNLlama previously defaulted to using FP32, but now default to using FP16 as well. Usage: C++ example:
Python API:
Python demo.py: