turboderp exllamav2 issues

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

MIT License

3.19k stars 234 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Trouble converting Phi-3-small

#527 dnhkng opened 17 hours ago
2
Help me.i cuda error.

#525 lugangqi opened 1 day ago
11
help me,error

#524 lugangqi opened 1 day ago
3
feature request: Radix Cache

#523 isamu-isozaki closed 3 days ago
2
Issue with flash attention when upgrading to torch 2.3.1

#522 waterangel91 closed 3 days ago
1
Lazy loading of 2 models give CUDA out of memory

#521 waterangel91 closed 2 days ago
4
Flask based paged attention with streaming and generator queue to dynamically add and remove jobs.

#520 rjmehta1993 opened 6 days ago
2
Question on Async generator

#519 waterangel91 closed 1 week ago
6
Problem running dynamic Gen.py in examples

#518 rjmehta1993 closed 6 days ago
2
There was a problem changing characters in the SillyTavern

#517 1Q18LAqakl opened 1 week ago
21
Dumb quantize/selective recompile/recapitation?

#516 tau0-deltav opened 1 week ago
1
Chameleon support

#515 end-me-please opened 1 week ago
2
inference_json Example not working

#513 rjmehta1993 closed 1 week ago
3
Support for architecture DeepseekV2ForCausalLM

#512 RodriMora opened 1 week ago
10
test_inference.py: --low_mem is broken unless --max_output_len is also set

#511 IMbackK closed 1 week ago
1
ROCM: Issues with wave64 device

#510 IMbackK closed 1 week ago
24
There is a problem when quantizing Qwen2

#509 1Q18LAqakl opened 2 weeks ago
9
extremely high CPU usage

#508 sfttfs closed 2 weeks ago
10
Quick question on Dynamic Generation

#507 dnhkng closed 2 weeks ago
2
Increase GPU utilization?

#506 sunflower-leaf closed 2 weeks ago
4
importing exllamav2.generator stops here

#505 lovebeatz opened 2 weeks ago
21
Covert.py measurement "Killed"

#504 GHBigD opened 2 weeks ago
6
Floating point exception when context length > chunk_size

#503 cikkle opened 2 weeks ago
7
RuntimeError: FlashAttention only supports Ampere GPUs or newer - despite asking it to disable flash_attention

#502 quarterturn closed 1 day ago
1
return last state in dynamic generator

#501 npott9 opened 2 weeks ago
0
Q-Cache - Token Generation Speed

#499 Vhallo opened 3 weeks ago
4
Problems with quantization and qwen2 inference

#498 nikitabalakin closed 3 weeks ago
4
Update Actions

#497 bdashore3 closed 3 weeks ago
0
Running humaneval against llama-3-8b-instruct exl2 quant results in a silent OOM when samples per task > 7

#496 LlamaEnjoyer opened 3 weeks ago
6
"Loading exllamav2_ext extension (JIT)... Building C++/CUDA extension" hangs forever

#495 AgeOfAlgorithms closed 3 weeks ago
3
EXL2 format spec?

#494 polarathene closed 2 weeks ago
3
Qwen 2 inference problem

#493 Sadeghi85 closed 2 weeks ago
15
Make comments in the readme look better

#491 CyberTimon closed 3 weeks ago
0
After updating exllamav2 to 0.1.0, the text generated by exui will not be pushed verbatim

#490 xldistance opened 3 weeks ago
1
Qauntization in glm4-9b failed

#489 Orion-zhen opened 3 weeks ago
3
Added steps to benchmark in README

#488 RodriMora closed 3 weeks ago
1
Added steps to benchmark using the included mmlu script

#487 RodriMora closed 3 weeks ago
0
LM Enforcer cause hanged generation and what is the Sampler setting

#486 waterangel91 closed 3 weeks ago
17
v0.1.3 lm format enforcer broken

#485 waterangel91 closed 4 weeks ago
2
Non stop generation after update to v0.1.1.0 and latest flash attention

#484 waterangel91 closed 4 weeks ago
6
v0.1.1 multi-gpu issue (fine in v0.0.21)

#483 surenchl closed 4 weeks ago
1
Quick, Non-Data-Driven Quantization

#482 alexbrowngh closed 2 weeks ago
3
module 'exllamav2_ext' has no attribute 'count_match'

#481 abpani closed 2 weeks ago
1
Can I not use flash attention? Because the model needs to be deployed to nvidia T4

#480 vikotse closed 2 weeks ago
4
Support MiniCPM architecture

#479 meigami0 closed 2 weeks ago
5
[feature request] LLAMA.CPP

#476 0wwafa closed 1 month ago
3
Problem with blinker...

#475 0wwafa opened 1 month ago
3
Phi-3 medium generation issue

#474 rjmehta1993 opened 1 month ago
3
quantization fails while writing shards

#472 theyunt closed 1 month ago
2
What does the implemention of ’segmenting input‘ in the exllamav2 called？

#470 laoda513 closed 1 month ago
3