bashlk
commented
3 days ago I am running into the same issue, also on Debian 12, on an older Intel CPU, while trying to run a Qwen2.5 exl2 model over the Open AI API (With cline and aider). In my case a few requests work and then it encounters this error after which the responses contain no/few characters. Unloading and loading the model again doesn't seem to help.
I'm running web-ui directly on physical hardware. I tried upgrading all the packages in my system which brought in a new kernel version but nothing changed after the upgrade.
Logs
13:27:44-948132 INFO Starting Text generation web UI
13:27:44-952671 WARNING
You are potentially exposing the web UI to the entire
internet without any access password.
You can create one with the "--gradio-auth" flag like
this:
--gradio-auth username:password
Make sure to replace username:password with your own.
13:27:44-954803 INFO Loading the extension "openai"
13:27:45-089753 INFO OpenAI-compatible API URL:
http://0.0.0.0:5000
Running on local URL: http://0.0.0.0:7860
13:27:51-237158 INFO Loading
"bartowski_Qwen2.5-Coder-14B-Instruct-exl2_4_25"
/home/gradio/text-generation-webui/installer_files/env/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:600: UserWarning: `do_sample` is set to `False`. However, `min_p` is set to `0.0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `min_p`.
warnings.warn(
13:27:58-621663 INFO Loaded "bartowski_Qwen2.5-Coder-14B-Instruct-exl2_4_25"
in 7.38 seconds.
13:27:58-623069 INFO LOADER: "ExLlamav2_HF"
13:27:58-624496 INFO TRUNCATION LENGTH: 8000
13:27:58-625390 INFO INSTRUCTION TEMPLATE: "Custom (obtained from model
metadata)"
^[[A/home/gradio/text-generation-webui/installer_files/env/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:590: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
warnings.warn(
/home/gradio/text-generation-webui/installer_files/env/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:600: UserWarning: `do_sample` is set to `False`. However, `min_p` is set to `0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `min_p`.
warnings.warn(
Output generated in 4.25 seconds (21.15 tokens/s, 90 tokens, context 896, seed 2117395925)
Output generated in 2.80 seconds (22.11 tokens/s, 62 tokens, context 1011, seed 1939351019)
Traceback (most recent call last):
File "/home/gradio/text-generation-webui/modules/text_generation.py", line 410, in generate_reply_HF
new_content = get_reply_from_output_ids(output, state, starting_from=starting_from)
File "/home/gradio/text-generation-webui/modules/text_generation.py", line 271, in get_reply_from_output_ids
reply = decode(output_ids[starting_from:], state['skip_special_tokens'] if state else True)
File "/home/gradio/text-generation-webui/modules/text_generation.py", line 181, in decode
return shared.tokenizer.decode(output_ids, skip_special_tokens=skip_special_tokens)
File "/home/gradio/text-generation-webui/installer_files/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 4004, in decode
return self._decode(
File "/home/gradio/text-generation-webui/installer_files/env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 654, in _decode
text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
OverflowError: out of range integral type conversion attempted
Output generated in 2.30 seconds (21.29 tokens/s, 49 tokens, context 1274, seed 1253585262)
Traceback (most recent call last):
File "/home/gradio/text-generation-webui/modules/text_generation.py", line 410, in generate_reply_HF
new_content = get_reply_from_output_ids(output, state, starting_from=starting_from)
File "/home/gradio/text-generation-webui/modules/text_generation.py", line 271, in get_reply_from_output_ids
reply = decode(output_ids[starting_from:], state['skip_special_tokens'] if state else True)
File "/home/gradio/text-generation-webui/modules/text_generation.py", line 181, in decode
return shared.tokenizer.decode(output_ids, skip_special_tokens=skip_special_tokens)
File "/home/gradio/text-generation-webui/installer_files/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 4004, in decode
return self._decode(
File "/home/gradio/text-generation-webui/installer_files/env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 654, in _decode
text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
OverflowError: out of range integral type conversion attempted
Output generated in 0.74 seconds (1.36 tokens/s, 1 tokens, context 1153, seed 8088406)
13:32:10-609612 INFO Loading
"bartowski_Qwen2.5-Coder-14B-Instruct-exl2_4_25"
13:32:16-405005 INFO Loaded "bartowski_Qwen2.5-Coder-14B-Instruct-exl2_4_25"
in 5.79 seconds.
13:32:16-407239 INFO LOADER: "ExLlamav2_HF"
13:32:16-408041 INFO TRUNCATION LENGTH: 8000
13:32:16-408885 INFO INSTRUCTION TEMPLATE: "Custom (obtained from model
metadata)"
Traceback (most recent call last):
File "/home/gradio/text-generation-webui/modules/text_generation.py", line 410, in generate_reply_HF
new_content = get_reply_from_output_ids(output, state, starting_from=starting_from)
File "/home/gradio/text-generation-webui/modules/text_generation.py", line 271, in get_reply_from_output_ids
reply = decode(output_ids[starting_from:], state['skip_special_tokens'] if state else True)
File "/home/gradio/text-generation-webui/modules/text_generation.py", line 181, in decode
return shared.tokenizer.decode(output_ids, skip_special_tokens=skip_special_tokens)
File "/home/gradio/text-generation-webui/installer_files/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 4004, in decode
return self._decode(
File "/home/gradio/text-generation-webui/installer_files/env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 654, in _decode
text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
OverflowError: out of range integral type conversion attempted
Output generated in 1.30 seconds (0.77 tokens/s, 1 tokens, context 1176, seed 304963644)lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 36 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz
CPU family: 6
Model: 58
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Stepping: 9
CPU(s) scaling MHz: 42%
CPU max MHz: 3800.0000
CPU min MHz: 1600.0000
BogoMIPS: 6799.95
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mm
x fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_go
od nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est
tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rd
rand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid
fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 128 KiB (4 instances)
L1i: 128 KiB (4 instances)
L2: 1 MiB (4 instances)
L3: 6 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-3
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: KVM: Mitigation: VMX disabled
L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT disabled
Mds: Mitigation; Clear CPU buffers; SMT disabled
Meltdown: Mitigation; PTI
Mmio stale data: Unknown: No mitigations
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS
Not affected; BHI Not affected
Srbds: Vulnerable: No microcode
Tsx async abort: Not affectedfree -m
total used free shared buff/cache available
Mem: 23982 1365 12767 4 10200 22617
Swap: 7999 0 7999nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3060 On | 00000000:01:00.0 Off | N/A |
| 0% 35C P8 11W / 170W | 181MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 777 G /usr/lib/xorg/Xorg 167MiB |
| 0 N/A N/A 964 G /usr/bin/gnome-shell 8MiB |
+---------------------------------------------------------------------------------------+uname -a
Linux bash-3lpc 6.1.0-27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01) x86_64 GNU/Linuxpython3 --version
Python 3.11.2conda --version
conda 23.5.2cat /etc/debian_version
12.8git rev-parse HEAD
cc8c7ed2093cbc747e7032420eae14b5b3c30311
jrruethe
Describe the bug
When running inference over openAI compatable API with Perplexica or avante.nvim the error sometimes appears, after that happnes it doesn't work anymore until I restart the program. (It worked fine with Open WebUI)
Is there an existing issue for this?
Reproduction
Screenshot
Logs
System Info