-
Hi there!
First of all, I appreciate the team for putting in the work for this research paper. I would like to preface this by saying that my comments here should just be considered a point of disc…
-
Hello everyone, I implemented save index into folder local, store in my laptop. Then, I could load them again. However, I have to struggle on load them because it can not find `docstore`, `index_store…
-
My reproduction of the results on location 9 of the NQ dataset in the longllmlingua paper using the prompt compressor resulted in a large discrepancy from the original results. My hyperparameters are …
-
## Description
I tried to quote the following documents directly,tools/pytorch-quantization/pytorch_quantization/calib/histogram.py,and Use HistogramCalibrator.compute_amax() to calculate the max…
-
https://github.com/vllm-project/llm-compressor/blob/5eddaa0d9adea0be6c8fd5cb5c14fbe017413f41/src/llmcompressor/transformers/compression/helpers.py#L202
-
Hi,
i try your example from the main page:
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
wget https://huggingface.co/neuralmagic/TinyLlama-1.1B-Chat-v…
-
### Your current environment
```text
PyTorch version: 2.3.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.3 LTS (x86_64)
GCC ve…
-
### 是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?
- [X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions
### 该问题是否在FAQ中有解答? | Is there an existing ans…
-
### Motivation.
There are more and more use cases, where we need to transfer KV caches between vLLM instances, or store KV caches for future use. Some concrete use cases:
- Disaggregated prefilling.…
-
### Describe the bug
MessageTokenLimiter works as expected for output not generated by Tools.
However, when Tools are used, this appears not to be the case.
### Steps to reproduce
- Create a custo…