search

harupy / mlflow

Open source platform for the machine learning lifecycle
https://mlflow.org
Apache License 2.0
0 stars 1 forks source link

a #79

Open harupy opened 1 year ago

harupy commented 1 year ago

https://chat.openai.com/share/b2b3a55a-31a8-413f-af5f-4070b42710c5

harupy commented 10 months ago 
sequenceDiagram
    participant C as Client
    participant M as MLflow<br>AI Gateway
    participant L as LLM provider<br>(e.g. OpenAI)

    C->>M: `gateway/chat/invocations<br>with {"stream": true}`
    M->>M: tranlslate request payload
    M->>L: `api.openai.com/v1/chat/completions<br>with {"stream": true}`
    L->>M: 'data: {"choices": ["hello"]}`<br>(may not be SSE)
    M->>M: translate stream into<br>SSE and common structure
    M->>C: 'data: {"candidates": ["hello"]}'<br>(SSE)

    L->>M: 'data: {"choices": ["..."]}`
    M->>C: 'data: {"candidates": ["..."]}`