Open harupy opened 1 year ago
sequenceDiagram
participant C as Client
participant M as MLflow<br>AI Gateway
participant L as LLM provider<br>(e.g. OpenAI)
C->>M: `gateway/chat/invocations<br>with {"stream": true}`
M->>M: tranlslate request payload
M->>L: `api.openai.com/v1/chat/completions<br>with {"stream": true}`
L->>M: 'data: {"choices": ["hello"]}`<br>(may not be SSE)
M->>M: translate stream into<br>SSE and common structure
M->>C: 'data: {"candidates": ["hello"]}'<br>(SSE)
L->>M: 'data: {"choices": ["..."]}`
M->>C: 'data: {"candidates": ["..."]}`
https://chat.openai.com/share/b2b3a55a-31a8-413f-af5f-4070b42710c5