如何使用byzerllm deploy部署嵌入及重排模型

chat 模型可以试用一下命令部署：

byzerllm deploy  --pretrained_model_type saas/official_openai \
--cpus_per_worker 0.01 \
--gpus_per_worker 0 \
--num_workers 1 \
--infer_params saas.api_key=xxxxx saas.model=llama2  saas.base_url="http://localhost:11434/v1/" \
--model ollama_llama2_chat

嵌入及重排及重排呢？部署了嵌入模型，测试：byzerllm query --model bge-m3 --query 你好，遇到异常：

2024-09-09 15:21:50,372 ERROR serialization.py:425 -- Failed to unpickle serialized exception
Traceback (most recent call last):
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/exceptions.py", line 50, in from_ray_exception
    return pickle.loads(ray_exception.serialized_exception)
TypeError: APIStatusError.__init__() missing 2 required keyword-only arguments: 'response' and 'body'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/_private/serialization.py", line 423, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/_private/serialization.py", line 305, in _deserialize_object
    return RayError.from_bytes(obj)
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/exceptions.py", line 44, in from_bytes
    return RayError.from_ray_exception(ray_exception)
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/exceptions.py", line 53, in from_ray_exception
    raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception
Traceback (most recent call last):
  File "/data/miniconda3/envs/byzer/bin/byzerllm", line 8, in <module>
    sys.exit(main())
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/byzerllm/byzerllm_command.py", line 143, in main
    resp = llm_client.chat_oai(
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/byzerllm/utils/client/byzerllm_client.py", line 1212, in chat_oai
    res = self._query(model, v)
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/byzerllm/utils/client/byzerllm_client.py", line 1886, in _query
    raise inst
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/byzerllm/utils/client/byzerllm_client.py", line 1876, in _query
    res = ray.get(worker.async_apply.remote(new_input_value))
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/_private/worker.py", line 2661, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/_private/worker.py", line 873, in get_objects
    raise value
ray.exceptions.RaySystemError: System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
  File "/data/miniconda3/envs/byzer/lib/python3.10/site-packages/ray/exceptions.py", line 50, in from_ray_exception
    return pickle.loads(ray_exception.serialized_exception)
TypeError: APIStatusError.__init__() missing 2 required keyword-only arguments: 'response' and 'body'

allwefantasy / byzer-llm

如何使用byzerllm deploy部署嵌入及重排模型 #43