Open MahmoudAshraf97 opened 1 month ago
@MahmoudAshraf97 I am investigating the issue now and would update here later.
Reproduced with 0.12.0.dev2024071600
Reproduced with
0.12.0.dev2024071600
@MahmoudAshraf97 Yeah, the fixed codes have not been merged into main yet. Let me tell you here once it got merged.
System Info
Who can help?
@byshiue
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Build using the official example instructions and switch
remove_input_padding
andpaged_kv_cache
toenable
then load the model using the class in
run.py
Expected behavior
The model should load fine
actual behavior
additional notes
I build with kv cache enabled to use in-flight batching, it's not in a usable state for now but this is for another issue check #1909