sophosympatheia
commented
6 months ago Here is a config.json diff between one model that has the issue and similar model that doesn't have the issue. The only difference is that the one on the right, the one with the issue, includes a quantization_config block.
{ {
"_name_or_path": "midnight-miqu-70b-v1.5", | "_name_or_path": "/home/llm/mergequant/models/BASE/152334
"architectures": [ | "architectures": [
"LlamaForCausalLM" | "LlamaForCausalLM"
], | ],
"attention_bias": false, | "attention_bias": false,
"attention_dropout": 0.0, | "attention_dropout": 0.0,
"bos_token_id": 1, | "bos_token_id": 1,
"eos_token_id": 2, | "eos_token_id": 2,
"hidden_act": "silu", | "hidden_act": "silu",
"hidden_size": 8192, | "hidden_size": 8192,
"initializer_range": 0.02, | "initializer_range": 0.02,
"intermediate_size": 28672, | "intermediate_size": 28672,
"max_position_embeddings": 32764, | "max_position_embeddings": 32764,
"model_type": "llama", | "model_type": "llama",
"num_attention_heads": 64, | "num_attention_heads": 64,
"num_hidden_layers": 80, | "num_hidden_layers": 80,
"num_key_value_heads": 8, | "num_key_value_heads": 8,
"pad_token_id": 0, | "pad_token_id": 0,
"pretraining_tp": 1, | "pretraining_tp": 1,
"rms_norm_eps": 1e-05, | "rms_norm_eps": 1e-05,
"rope_scaling": null, | "rope_scaling": null,
"rope_theta": 1000000, | "rope_theta": 1000000,
"tie_word_embeddings": false, | "tie_word_embeddings": false,
"torch_dtype": "float16", | "torch_dtype": "float16",
"transformers_version": "4.36.2", | "transformers_version": "4.36.2",
"use_cache": true, | "use_cache": true,
"vocab_size": 32000 | "vocab_size": 32000,
} | "quantization_config": {
> "quant_method": "exl2",
> "version": "0.0.15",
> "bits": 5.0,
> "head_bits": 6,
> "calibration": {
> "rows": 100,
> "length": 2048,
> "dataset": "(default)"
> }
> }
> }
Ph0rk0z
RedDragonGecko
oldgithubman
Goldenkoron
alxfoster
Describe the bug
Within the past week, I've noticed textgen webui sometimes ignores my GPU split string when loading a model with either ExLlamav2_HF or ExLlamav2. It's not a consistent issue across all models but it is consistent for the models that have the issue. I think so far the models with issues have all been 70B or 103B Miqu merges.
I have made sure that both GPUs are visible to the OS and neither of them are filtered through CUDA_VISIBLE_DEVICES or other directives that would stop textgen from using both cards.
What seems to happen is textgen ignores the second GPU when loading the model. Say I specify a 20,24 GPU split. Textgen should load up 20 GB into Card 0 and then load up the rest into Card 1--typical behavior. Instead, it maxes out Card 0 until it triggers an OOM error without touching Card 1.
I am running the latest version of textgen (git pull shows all up to date) and its dependencies specified in requirements.txt.
I do set
PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsyncprior to running server.py, but I have been doing that for about a month without this issue appearing until recently.Is there an existing issue for this?
Reproduction
Attempt to load a Exllamav2 model on a system with two or more GPUs. Textgen ignores the specified GPU split for some models.
Screenshot
No response
Logs
System Info