-
Hello. Thank for providing vLLM as a great open-source tool for inference and model serving! I was able to build vLLM on a cluster I maintain, but it only appears to work on a single MI210 GPU. Can so…
-
Greetings, @cipher982!
I've seen the benchmark application https://www.llm-benchmarks.com/local and it looks great! I'm currently working on a competitive analysis of this 4 backends: Transformers…
-
Hi, can you please provide a guide or support to use local llm models like Ollama lama3.1 8b or 70b
-
As per title.
Example: with GPUs like 3060 12GB or 3090 24GB.
-
## What
Let's support shape inference for operators whose inputs have dynamic shape.
I've made a list of operators to support dynamic-shaped LLM inference.
### First milestone (for token gen …
-
Would be great if someone could look into implementing this particular version of int8 for serving LLMs.
https://huggingface.co/blog/hf-bitsandbytes-integration
-
When you pull this docker file:intelanalytics/ipex-llm-serving-xpu:2.1.0 and start vllm serving like the following cmd:
python -m ipex_llm.vllm.xpu.entrypoints.openai.api_server \
--served-model-n…
-
when i use the example in multimodel, i download the original model-liuhaotian/llava-v1.5-7b,but some error occur?
llama = from_hugging_face(
File "/usr/local/lib/python3.10/dist-packages/tensor…
-
- [ ] [LoRA Land: Fine-Tuned Open-Source LLMs that Outperform GPT-4 - Predibase - Predibase](https://predibase.com/blog/lora-land-fine-tuned-open-source-llms-that-outperform-gpt-4)
# LoRA Land: Fine…
-
### 🚀 The feature
Right now mainly proprietary LLMs are supported. Would be great to also support DIY/OSS LLMs - for instance, hosted in [Databricks Model Serving](https://docs.databricks.com/en/mach…