-
I would like to know if there is a way to get usage statistics with each request (maybe with a flag parameter):
I would like to know queue wait time, num_prompt_tokens, num_generated_tokens, time…
-
Suppose we want to load-test an API which uses server-sent events (SSE). Is it possible to measure the time-to-first-byte using Goose?
-
### Describe the bug
`interpreter --local
:88: SyntaxWarning: "is" with a literal. Did you mean "=="?
▌ Open Interpreter is compatible with several local model providers.
[?] What one would yo…
-
## What are the problems?(screenshots or detailed error messages)
想问下有性能分析的工具嘛?profiler相关,还是只能用nsight profile这种自己去看一些算子性能
## What are the types of GPU/CPU you are using?
GPU:A100-80G-SXM4
## What…
-
The current version requires an Internet connection to download the models when it is first used after deployment.
Will the future add ways to deploy without the Internet? This will make LibrePhoto…
-
I wanted to try a self-hosted offline LLM to summarize some notes and this looked like the most straightforward thing to try! I've attached a screenshot with hopefully relevant information
Tutorial…
-
I have been running the scripts from [https://docs.vllm.ai/en/latest/models/spec_decode.html](https://docs.vllm.ai/en/latest/models/spec_decode.html ) on how to do speculative decoding with vLLM.
H…
-
# Proposed Feature
Add an efficient interface for generation probabilities on fixed prompt and completion pairs. For example:
```python
# ... load LLM or engine
prompt_completion_pairs = [
…
-
The newest drivers are in use, the system is a Ryzen 2700x CPU with 16GB of RAM and a 16GB A770 GPU on Windows 11.
The instructions in the docs were followed precisely.
Upon attempting to execut…
-
### Question Validation
- [X] I have searched both the documentation and discord for an answer.
### Question
I don't use huggingface because of proxy issue. So, I tried to make local embeddin…