Open WangxuP opened 2 months ago
我在使用vllm的 /v1/chat/completions 接口的时候,是可以正常使用的,而且速度是要比xinference快。
这个问题是xoscar库的问题,目前已经合并到0.3.2版本 https://github.com/xorbitsai/xoscar/pull/87
使用 pip install xoscar==0.3.2
升级后再压测试试
这个问题是xoscar库的问题,目前已经合并到0.3.2版本 xorbitsai/xoscar#87 使用
pip install xoscar==0.3.2
升级后再压测试试
升级了好像还是有问题,报错信息基本一致,好像是 stream == True
之后就会触发这个BUG
这个问题是xoscar库的问题,目前已经合并到0.3.2版本 xorbitsai/xoscar#87 使用
pip install xoscar==0.3.2
升级后再压测试试升级了好像还是有问题,报错信息基本一致,好像是
stream == True
之后就会触发这个BUG
升级后重启xinference了吗?粘贴下报错日志看看
这个问题是xoscar库的问题,目前已经合并到0.3.2版本 xorbitsai/xoscar#87 使用
pip install xoscar==0.3.2
升级后再压测试试升级了好像还是有问题,报错信息基本一致,好像是
stream == True
之后就会触发这个BUG升级后重启xinference了吗?粘贴下报错日志看看
2024-07-19 15:01:38,014 transformers.models.llama.modeling_llama 63561 WARNING We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
2024-07-19 15:01:49,005 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 3.22 tokens/s.
2024-07-19 15:01:50,378 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 20.42 tokens/s.
2024-07-19 15:01:50,871 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 15.04 tokens/s.
2024-07-19 15:02:03,688 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 25.22 tokens/s.
2024-07-19 15:02:18,889 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36816) during chat.
2024-07-19 15:02:24,799 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 0.85 tokens/s.
2024-07-19 15:02:29,739 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 1.02 tokens/s.
2024-07-19 15:02:33,928 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36870) during chat.
2024-07-19 15:02:33,939 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36872) during chat.
2024-07-19 15:02:33,951 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36874) during chat.
2024-07-19 15:02:33,955 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36876) during chat.
2024-07-19 15:02:33,963 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36864) during chat.
2024-07-19 15:02:33,978 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36884) during chat.
2024-07-19 15:02:33,983 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36888) during chat.
2024-07-19 15:02:33,987 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36886) during chat.
版本是
xinference 0.13.1
xoscar 0.3.2
这个问题是xoscar库的问题,目前已经合并到0.3.2版本 xorbitsai/xoscar#87 使用
pip install xoscar==0.3.2
升级后再压测试试升级了好像还是有问题,报错信息基本一致,好像是
stream == True
之后就会触发这个BUG升级后重启xinference了吗?粘贴下报错日志看看
2024-07-19 15:01:38,014 transformers.models.llama.modeling_llama 63561 WARNING We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache) 2024-07-19 15:01:49,005 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 3.22 tokens/s. 2024-07-19 15:01:50,378 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 20.42 tokens/s. 2024-07-19 15:01:50,871 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 15.04 tokens/s. 2024-07-19 15:02:03,688 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 25.22 tokens/s. 2024-07-19 15:02:18,889 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36816) during chat. 2024-07-19 15:02:24,799 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 0.85 tokens/s. 2024-07-19 15:02:29,739 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 1.02 tokens/s. 2024-07-19 15:02:33,928 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36870) during chat. 2024-07-19 15:02:33,939 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36872) during chat. 2024-07-19 15:02:33,951 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36874) during chat. 2024-07-19 15:02:33,955 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36876) during chat. 2024-07-19 15:02:33,963 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36864) during chat. 2024-07-19 15:02:33,978 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36884) during chat. 2024-07-19 15:02:33,983 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36888) during chat. 2024-07-19 15:02:33,987 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36886) during chat.
版本是
xinference 0.13.1 xoscar 0.3.2
换成vllm引擎呢,之前InvalidStateError: invalid state之后,会导致整个接口挂掉,无法继续响应任何请求,即使推理引擎还是正常的。
This issue is stale because it has been open for 7 days with no activity.
这个问题是xoscar库的问题,目前已经合并到0.3.2版本 xorbitsai/xoscar#87 使用
pip install xoscar==0.3.2
升级后再压测试试升级了好像还是有问题,报错信息基本一致,好像是
stream == True
之后就会触发这个BUG升级后重启xinference了吗?粘贴下报错日志看看
2024-07-19 15:01:38,014 transformers.models.llama.modeling_llama 63561 WARNING We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache) 2024-07-19 15:01:49,005 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 3.22 tokens/s. 2024-07-19 15:01:50,378 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 20.42 tokens/s. 2024-07-19 15:01:50,871 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 15.04 tokens/s. 2024-07-19 15:02:03,688 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 25.22 tokens/s. 2024-07-19 15:02:18,889 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36816) during chat. 2024-07-19 15:02:24,799 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 0.85 tokens/s. 2024-07-19 15:02:29,739 xinference.model.llm.pytorch.utils 63561 INFO Average generation speed: 1.02 tokens/s. 2024-07-19 15:02:33,928 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36870) during chat. 2024-07-19 15:02:33,939 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36872) during chat. 2024-07-19 15:02:33,951 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36874) during chat. 2024-07-19 15:02:33,955 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36876) during chat. 2024-07-19 15:02:33,963 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36864) during chat. 2024-07-19 15:02:33,978 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36884) during chat. 2024-07-19 15:02:33,983 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36888) during chat. 2024-07-19 15:02:33,987 xinference.api.restful_api 63191 INFO Disconnected from client (via refresh/close) Address(host='127.0.0.1', port=36886) during chat.
版本是
xinference 0.13.1 xoscar 0.3.2
换成vllm引擎呢,之前InvalidStateError: invalid state之后,会导致整个接口挂掉,无法继续响应任何请求,即使推理引擎还是正常的。
用vllm引擎,出现同样问题。请问有什么解决方案吗?
Describe the bug
我们在压测xinference时候发现,V100 2卡,调用/v1/chat/completions接口,stream参数是True,模型用qwen-14b-chat,用jmeter10并发进行压测,压测1分钟xinference就挂了,如果stream是False,是可以的.
报错日志
requtirements.txt
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.