Use Single HTTP API for Robust, Generalizable Chat Streaming

[debanjum]

Stream Status Messages via Streaming Response from server to web client

Overview

Use a single API controller for advanced chat streaming and non-streaming mode. Improve code maintainability and generalize to other Khoj client apps.

Update web client to use the streamed response to show train of thought, stream response and render references.

Motivation

• Using a single chat API to simplify code, increase dev velocity
• GET API will allow streaming Khoj train of thought, references etc. in a structured form from server to Khoj client apps
  • So streaming Khoj's train of thought and other status updates to clients from server should now be possible
  • Passing auth headers from client apps is possible
    • Websockets don't allow passing request headers
  • Don't need to pass references via ### compiled references: custom delimiter
• Khoj chat doesn't need bi-directional syncing yet. So using HTTP stream to send updates should be good enough

Details

Major

• [X] Stream Khoj's train of thought, response via simple HTTP streaming
• [X] Reduce to single chat API controller for streaming and non-streaming chat response
• [X] Update Clients to use new HTTP streaming to render train of thought, extract references, stream response
  • [X] Update web client to use new streaming to render train of thought, extract references, stream response
  • [X] Start streaming Khoj train of thought to Khoj Obsidian plugin using new streaming mode
  • [X] Start streaming Khoj train of though to Khoj Desktop Clients using new streaming mode
  • No updates to Whatsapp, Emacs required as they use non-streaming response, which has the original interface

    Minor

• [X] Save message to conversation history even if client disconnects
• [X] Stream references in a structured form via HTTP streaming instead of using ### compiled references: separator
• [X] Log total response time and time to first token for overall chat latency analysis