search

Lightning-AI / LitServe

Lightning-fast serving engine for any AI model of any size. Flexible. Easy. Enterprise-scale.
https://lightning.ai/docs/litserve
Apache License 2.0
2.51k stars 160 forks source link

Update media type and JSON handling in OpenAISpec #360

Closed fabigr8 closed 2 weeks ago

fabigr8 commented 2 weeks ago

To comply with actual OpenAI API streaming responses i changed the content type and the json handling in the streaming implementation of openAI-spec see #359

Before submitting - [X] Was this discussed/agreed via a Github issue? (no need for typos and docs improvements) - [X] Did you read the [contributor guideline](https://github.com/Lightning-AI/pytorch-lightning/blob/main/.github/CONTRIBUTING.md), Pull Request section? - [X] Did you make sure to update the docs? --> Not applicable - [X] Did you write any new necessary tests? --> not necessary

How does this PR impact the user? As a user, I want to use LitServe to connect to third-party libraries or applications (such as Open WebUI) via openAI streaming API. This PR changes the LitServe OpenAI-spec to fully comply with the OAI streaming API and solve existiing connection issues.

What does this PR do?

Fixes #359

PR review

Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

YES 💯

codecov[bot] commented 2 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 96%. Comparing base (719e037) to head (54177b4). Report is 1 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #360 +/- ## =================================== Coverage 96% 96% =================================== Files 22 22 Lines 1408 1408 =================================== Hits 1345 1345 Misses 63 63 ```