ajndkr / lanarky

The web framework for building LLM microservices
https://lanarky.ajndkr.com/
MIT License
976 stars 74 forks source link

feat(core): add `EventSource` protocol #150

Closed ajndkr closed 11 months ago

ajndkr commented 11 months ago

Description

Fixes #121 Fixes #112

This PR adds the EventSource protocol for streaming LLM output. We still respect the text and JSON streaming modes but all events will now stream as follows:

data: <llm_token_1>

data: <llm_token_2>

This also allows the possibility of implementing named events for special use cases:

event: <event_a>
data: <llm_token_1>

event: <event_b>
data: <llm_token_2>

Finally, the StreamingResponse also improves error handling by sending a 500 error event when chain execution fails (Fixes #146)

Changelog: