Wolfsauge
commented
5 months ago Reproduced with latest versions of vLLM backend in podman (0.4.2) and model (meta-llama/Meta-Llama-3-8B-Instruct, commit c4a54320a52ed5f88b7a2f84496903ea4ff07b45, 2024-05-13) downloaded from hf with latest updates to the configs.
$ podman run \
--replace \
--device nvidia.com/gpu=all \
--name=vllm \
-e TRANSFORMERS_OFFLINE=1 -e HOST_IP=192.168.0.185 \
--ipc=host \
-dit \
-p 8000:8000 \
-v /models:/workspace/models \
docker.io/vllm/vllm-openai:v0.4.2 \
--model /workspace/models/meta-llama/Meta-Llama-3-8B-Instruct \
--served-model-name meta-llama/Meta-Llama-3-8B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--gpu-memory-utilization 0.95Example curl on the same endpoint:
$ curl -s http://mnemosyne.local:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3-8B-Instruct",
"messages": [
{
"role": "user",
"content": "Compose a poem that explains the concept of recursion in programming."
}
]
}'|jq .
{
"id": "cmpl-7dcf3aa6399a48a2ba42da95dfc748dc",
"object": "chat.completion",
"created": 1715720212,
"model": "meta-llama/Meta-Llama-3-8B-Instruct",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "What a fascinating topic! Here's a poem that attempts to explain recursion in programming:\n\nIn code, a function calls itself, it's true,\nA recursive dance, with loops anew.\nA method calls its own name, a clever trick,\nTo solve a problem, with a clever kick.\n\nIt's a function within a function, a nest,\nA recursive call, to pass the test.\nThe function calls itself, with parameters in tow,\nTo solve a problem, that's too complex to know.\n\nThe function calls itself, again and again,\nUntil the base case, is reached and sustained.\nThe recursion unwinds, like a coiled spring,\nUntil the solution, is finally brought to sing.\n\nThe function returns, with its value in hand,\nThe recursive call, is unwound, like a thread in a strand.\nThe stack is cleared, the memory is freed,\nThe recursive function, has solved the problem, indeed.\n\nRecursion's magic, is a wondrous thing,\nA way to solve problems, that would otherwise sting.\nIt's a clever tool, in a programmer's kit,\nTo tackle complexity, with a clever hit.\n\nSo here's to recursion, a programming delight,\nA way to solve problems, with a recursive might.\nIt's a concept that's tricky, but oh so grand,\nA way to solve problems, in a programming land.\n\nNote: I've used a few technical terms in the poem, such as \"function\", \"parameters\", \"base case\", and \"stack\", to make it more accurate and accessible to programmers. I hope it helps to explain the concept of recursion in a poetic way!"
},
"logprobs": null,
"finish_reason": "stop",
"stop_reason": 128009
}
],
"usage": {
"prompt_tokens": 23,
"total_tokens": 352,
"completion_tokens": 329
}
}Currently known workaround:
$ pip uninstall dspy-ai -y
Found existing installation: dspy-ai 2.4.9
Uninstalling dspy-ai-2.4.9:
Successfully uninstalled dspy-ai-2.4.9
$ pip install dspy-ai==2.4.7
Collecting dspy-ai==2.4.7
Using cached dspy_ai-2.4.7-py3-none-any.whl.metadata (36 kB)
[...]
WARNING: The candidate selected for download or install is a yanked version: 'dspy-ai' candidate (version 2.4.7 at https://files.pythonhosted.org/packages/b3/78/e383bf31195aa9ced4748b1a012d9be928dfda782e7abae02858898afa6f/dspy_ai-2.4.7-py3-none-any.whl (from https://pypi.org/simple/dspy-ai/) (requires-python:>=3.9))
Reason for being yanked: <none given>
Using cached dspy_ai-2.4.7-py3-none-any.whl (200 kB)
Installing collected packages: dspy-ai
Successfully installed dspy-ai-2.4.7
rakataprime
timchen0618
brando90
Using the dspy.HFClientVLLM method is broken since dsp version 2.4.9.
I have reproduced the error with the below script and the versions 2.4.5, 2.4.6 and 2.4.7 of dspy work as expected, while 2.4.8 doesn't seem to exist and 2.4.9 is broken.
I've been doing these tests with
Python 3.11.9.The erroneous run looks like this:
When it's working it looks like this:
Example script to reproduce the error:
Please note, the downside of using the dspy.OpenAI method (the commented section in the script above) as a workaround in 2.4.9, will result in only 1 request going on towards vLLM at any time. It's possible to use the openai-python module with a self-created httpx client, which can then have arbitrary settings for the connection pool size ... HTH