Closed HayeonLee closed 1 month ago
Hi @HayeonLee,
Thanks for your support with MInference.
pip install transformers==4.41.2
. We will look into how we can fix this issue in our package in the future.Hi @iofu728, those worked for me! Thank you for the valuable comments:D
Describe the issue
Hi, thank you for sharing the awewome work!
I am trying to reproduce the results in Table 5 and there are two problems.
1) MInference + SnapKV: I encountered 'RuntimeError: CUDA error: an illegal memory access was encountered.' So, I have a difficulty to reproduce the results
2) Additional question regarding attn_type: minference_with_dense. I thought it is the same model with flash attention version full attention, but it showed the poor results compared with full attention or attn_type: minference.
I used a single A100.
Could you please help me to fix this situattions? Any comments are welcome. Thank you!