-
I've been reading through the documentation and I couldn't get a 100% clear answer to my question. Apologies if I missed anything.
From what I gather, sigma requires an internet connection to work,…
-
**Checklist**
- [x] I made sure that there are no existing issues - open or closed - to which I could contribute my information.
- [x] I have read the FAQ and my problem isn't listed.
- [x] I …
-
I completed the installation of DistServe. When I tried to run the offline.py using my downloaded llama2 model, I encountered the following problem.
Traceback (most recent call last):
File "/hom…
-
Gemini Nano weights from Google Chrome are on [HuggingFace](https://huggingface.co/wave-on-discord/gemini-nano). You can run the inference using this model with [MediaPipe LLM inference](https://githu…
-
Failed to run python `offline_inference.py` from [link](https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/example/CPU/vLLM-Serving) for vLLM offline inference on CPU. It seems that `llm…
-
From time to time I receive an error message: "root : ERROR : No outline for {topic}. Will directly search with the topic."
I've modified one of the examples to use Ollama with the persona exam…
-
When I run the code "CUDA_VISIBLE_DEVICES=3 TRANSFORMERS_OFFLINE=1 lm_eval --model hf --model_args pretrained=/public/MountData/yaolu/LLM_pretrained/LLAMA2_7B/,trust_remote_code=True --tasks mmlu,cm…
-
### Proposal to improve performance
_No response_
### Report of performance regression
hi, sorry to trouble! I compared the performance of llama3-8b-instruct with mlp-speculator(llama3-8b-accelerat…
-
### 📚 The doc issue
The quantization example at https://docs.vllm.ai/en/latest/quantization/auto_awq.html can't be run - it looks like AWQ is looking for safetensors files and https://huggingface.c…
-
I would like to use techniques such as Multi-instance Support supported by the tensorrt-llm backend. In the documentation, I can see that multiple models are served using modes like Leader mode and …