search

64bit / async-openai

Rust library for OpenAI
https://docs.rs/async-openai
MIT License
1.11k stars 165 forks source link

Support vLLM-style emulation of OpenAI endpoint. #165

Closed MichaelMcCulloch closed 6 months ago

MichaelMcCulloch commented 9 months ago 
my_rust_project-1   | [2023-12-07T23:13:30Z ERROR async_openai::error] failed deserialization of: {"id": "cmpl-2fe481188f2447dca57fd0be25f8fafc", "model": "TheBloke/zephyr-7B-alpha-AWQ", "choices": [{"index": 0, "delta": {"role": "assistant"}, "finish_reason": null}]}
vllm-nv-1        | INFO 12-07 23:13:31 llm_engine.py:636] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%
my_rust_project-1   | [2023-12-07T23:13:31Z ERROR async_openai::error] failed deserialization of: {"id": "cmpl-2fe481188f2447dca57fd0be25f8fafc", "created": 76644, "model": "TheBloke/zephyr-7B-alpha-AWQ", "choices": [{"index": 0, "delta": {"content": "Acc"}, "finish_reason": null}]}

As you can see from the log, the created timestamp is not always present, and object never is. Now it's compatible with both.

64bit commented 8 months ago

Hi @MichaelMcCulloch

Thank you for the PR, The challenge here is maintaining compatibility with various providers. I'd rather stick with one provider - that is OpenAI - adding more creates maintenance challenges.

MichaelMcCulloch commented 8 months ago

I see what you are saying. But does making a particular field optional add maintenance challenges? How would you request that I improve the current pr?

MichaelMcCulloch commented 8 months ago

I see the entire types module has been refactored. So that answers my question. Would it be acceptable if I modified my pr to match?

64bit commented 6 months ago

Hey I had added scope tag and I missed to respond about your comments, sorry about that.

I see your point, however timeline-wise after your PR I had added section about this topic of what should be within the scope of this library https://github.com/64bit/async-openai?tab=readme-ov-file#contributing - so please know that your PR's contribution came in the form of documentation