-
followed installation of Vllm via this [link](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/vLLM_quickstart.html)
tried running via docker too: Here is the [image](https://hub.docke…
-
Hey all, I have a quick question, is onnxruntime-genai ([https://onnxruntime.ai/docs/genai/api/python.html](https://onnxruntime.ai/docs/genai/api/python.html)) supported in Triton Inference Server's O…
-
Running the benchmark script on a llama-3-8b-inst on inferentia 2 (djl-serving) results in:
```
python3.10 token_benchmark_ray.py \
--model "openai/llam…
-
Perhaps it's user error but I can't pass a custom OpenAI `base_url` to redirect the requests to a Databricks serving endpoint. This would be ideal for using {chattr} to interact with Databricks [found…
-
https://github.com/octoml/mlc-llm/blob/batch-serving/serve/mlc_serve/engine/engine_common.py#L385-L399
For streaming case, we cannot clamp the generated tokens and recompute them.
Moreover, since …
-
### What are you trying to do?
Currently, ollama supports most llms and I know that it also supports vision model. I thought it would be nice if the sound model could also be used through ollama. I w…
-
### Checked other resources
- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a sim…
-
ValueError: LoRA rank 64 is greater than max_lora_rank 16.
-
The current model is unable to calculate the time spent on first token and rest tokens, can we add this msg ?
-
### Your current environment
Using latest available docker image: vllm/vllm-openai:v0.5.0.post1
### 🐛 Describe the bug
I am getting as response "Internal Server Error" when calling the /v1/embedd…