-
I would like to use `llm` completely offline. I can install the `llm` python package offline with:
`pip download llm`
and then using this on the offline computer
`pip install --no-index --fin…
-
Hello,
first of all, a cool project!
Larger codebases often significantly exceed even the largest context windows available these days, while offline LLMs are even more troublesome in this regar…
-
### Your current environment
Referring to the issue #5181 "The Offline Inference Embedding Example Fails", the method LLM.encode() can only work for embedding models. Is there any idea to get the ou…
-
I'm attempting to export OTLP Traces to Dynatrace via the TRACELOOP_HEADERS environment variable that contains my API token.
I've tried setting the environment variable with TypeScript code, and th…
-
Hi, I am trying to run vllm-serving for the neural-chat model using https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/example/GPU/vLLM-Serving . However facing this issue
![image](htt…
-
**Is your feature request related to a problem? Please describe.**
As of now, Haystack's evaluators which extend LLMEvaluator only support OpenAI. I would like for support through llama.cpp to be add…
-
### Proposal to improve performance
Currently, vLLM allocates all available GPU memory after loading model weights, regardless of the max_model_len setting. This can lead to inefficient memory usage,…
-
For example: https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/example/GPU/PyTorch-Models/Model/qwen1.5/generate.py
The current inference output is generated all at once.
However, t…
-
Hi,
is there a way to test a guidance grammar without using a LLM call? I'm trying to write a complex grammar, and it is very cumbersome to develop it as LLM calls take their time. I would like to …
-
According to the installation instructions, an error occurred when I ran the offline benchmarking script and I have installed the sentencepiece. How can I solve it ? Thanks!
INFO 05-16 17:10:41 …