-
### System Info
Debian 11
`nvidia-smi`
```
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07 Driver Version: 535.16…
-
### vllm latest
I add some logger in /vllm/model_executor/models/llama.py ,I want to print the attention ,like that
if I start llm server,the error is
[rank0]: During handling of the above e…
-
### System Info
- CPU architecture: x86_64
- Host memory: 256GB
- GPU
+ Name: NVIDIA A30
+ Memory: 24GB
- Libraries
+ TensorRT-LLM: v0.11.0
+ TensorRT: 10.1.0
+ CUDA: 12.6
+ NVID…
-
### Your current environment
# Initialize the LLaVA-1.5 model
llm = LLM(model="llava-hf/llava-1.5-7b-hf")
print(llm)
#embed_last_hook = Hook(model.language_model.model.norm) # for save embed
…
-
The official ollama supports this model in v0.3.4
[https://github.com/ollama/ollama/releases/tag/v0.3.4](https://github.com/ollama/ollama/releases/tag/v0.3.4)
Tried with ollama in 2.1.0b20240820, …
-
**Describe the bug**
When using a standard RAG pipeline I get the above error.
**Error message**
```
File "/home/felix/PycharmProjects/anychat/src/anychat/analysis/rag.py", line 124, in qu…
-
### Discussed in https://github.com/ggerganov/llama.cpp/discussions/9228
Originally posted by **bulaikexiansheng** August 29, 2024
I try to use the speculative decoding script, the command is …
-
### Your current environment
```text
The output of `curl -X POST localhost:8000/v2/models/vllm_model/generate -d '{"text_input": "What is Triton Inference Server?", "parameters": {"stream": false,…
-
## Type of issue
- I conducted some benchmarks on Intel Core Ultra 7 155H about 3 months ago using this release: [b2568](https://github.com/ggerganov/llama.cpp/releases/tag/b2568), and these are th…
-
found intel-openmp in /home/miniforge3/envs/ipex-llm-langchain-chatchat/lib/libiomp5.so
found tcmalloc in /home/miniforge3/envs/ipex-llm-langchain-chatchat/lib/python3.11/site-packages/ipex_llm/lib…