-
Hi,
Thanks for the amazing work on streaming-llm. While reading the paper, I came up with this question on why applying "attention sink" also works with models with alibi position embedding.
One o…
-
Firstly, I'd like to express my appreciation for your insightful paper and the open-source 'streaming-llm'. Your approach and experiments are truly commendable. I hope you don't mind, I would really a…
-
### Have you searched for similar requests?
Yes
### Is your feature request related to a problem? If so, please describe.
llama.cpp has the feature to re-use the context window if the beginning of …
-
Nice work!
I am wondering whether this attention sink magic is still needed for LLMs that has been already trained with window attention (e.g. [mistral](https://github.com/mistralai/mistral-src)). …
-
### System Info
Google Colab with GPU T4 and CUDA 12.2.
TensorRT-LLM version: 0.9.0.dev2024040200.
Here is the [minimum reproducible notebook](https://colab.research.google.com/drive/1xAxZKYHx_Qq4g…
-
I tried naively to add examples in https://github.com/mit-han-lab/streaming-llm/blob/main/data/mt_bench.jsonl, including examples with length of 4k tokens, without changing anything in the script. I r…
-
Hi
https://colab.research.google.com/drive/1YtXE_JKVntkGK14Yo9thjCjPMVzhA71d?usp=sharing
Here is the colab, but it doesn't run in colab it stops after a while due to memory overload or something…
-
About int8_kv_cache I did some tests:
> Test model is mistral-7b
> My test inference code comes from `run.py`, supplementing runner.generate's time-consuming statistics,Added warm up code.
> Input…
-
### Describe the bug
Installation characteristics.
1. How long does it take to install the program?
2. How much free space is required for installation?
3. On which disks is it installed?
4. In…
-
`raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled`
Any plans to get Metal support for us M2 users without CUDA? Thanks!