UPDATE: Silly me, the Q4_K_M version barely fits in 80GB but only when loading the model with 8k context, which is a little surprising because I don't have issues with running 120b q3-q4 models with 32k context. Is the huge K/V cache in this particular case normal?
Original post:
(I'm not sure if that's a bug or newer Qwen models are not supported, I can see some kind of support added in #549 but it seems that was some kind of converted model)
I tried 2 GGUFs to make sure it's not a problem with corrupted file or me joining them wrong, I didn't check smaller Qwen models.
Q2 immediately allocates 80GB of ram and crashes, Q4 is loading with usual speed up to ~50-60GB ram usage and spits out error with loading model.
Q2 + 1.57.1 first loads model up to 40-50GB, then allocated additional 80GB, spits out warning and crashes after some time with this message:
WARNING: failed to allocate 82120.00 MB of pinned memory: out of memory
llama_kv_cache_init: CPU KV buffer size = 82120.00 MiB
llama_new_context_with_model: KV self size = 82120.00 MiB, K (f16): 41060.00 MiB, V (f16): 41060.00 MiB
llama_new_context_with_model: CUDA_Host input buffer size = 80.29 MiB
WARNING: failed to allocate 4569.40 MB of pinned memory: out of memory
GGML_ASSERT: D:\a\koboldcpp\koboldcpp\ggml-backend.c:555: data != NULL && "failed to allocate buffer"
UPDATE: Silly me, the Q4_K_M version barely fits in 80GB but only when loading the model with 8k context, which is a little surprising because I don't have issues with running 120b q3-q4 models with 32k context. Is the huge K/V cache in this particular case normal?
Original post: (I'm not sure if that's a bug or newer Qwen models are not supported, I can see some kind of support added in #549 but it seems that was some kind of converted model)
I tried 2 GGUFs to make sure it's not a problem with corrupted file or me joining them wrong, I didn't check smaller Qwen models.
Tested koboldcpp versions: 1.57.1 | 1.59.1 | 1.60 (cuda + windows) Repo: https://huggingface.co/Qwen/Qwen1.5-72B-Chat-GGUF Models:
Q2 immediately allocates 80GB of ram and crashes, Q4 is loading with usual speed up to ~50-60GB ram usage and spits out error with loading model. Q2 + 1.57.1 first loads model up to 40-50GB, then allocated additional 80GB, spits out warning and crashes after some time with this message:
Q4 files were joined with:
Koboldcpp config file:
Q2_K 1.60:
Q4_K_M 1.60: