The stream is not response as a

[joveth1]

Description

I'm using the stream to response gptchat result to front.

My Demo like :

func ChatWithGpt(c *gin.Context) {
    var req []UserMessages
    if err := c.BindJSON(&req); err != nil || len(req) == 0 {
        return
    }
    messages := make([]client.ChatMessage, 0)
    question := ""
    for index := range req {
        qa := req[index]
        if index == len(req)-1 {
            // the last text is user question
            if strings.TrimSpace(qa.UserText) != "" {
                question = strings.TrimSpace(qa.UserText)
            }
            break
        }
        if strings.TrimSpace(qa.UserText) != "" {
            messages = append(messages, client.ChatMessage{
                Role:    openai.ChatMessageRoleUser,
                Content: strings.TrimSpace(qa.UserText),
            })
        } else if strings.TrimSpace(qa.InitText) != "" {
            messages = append(messages, client.ChatMessage{
                Role:    openai.ChatMessageRoleAssistant,
                Content: strings.TrimSpace(qa.InitText),
            })
        }
    }
    log.Infof("msg: %v", messages)
    chanStream := make(chan client.ChatResponse, 500)
    stream, err := client.ChatCompletionWithStream(
        context.Background(),
        client.ChatRequest{
            History:           messages,
            TopK:              1,
            Temperature:       0.7,
            ScoreThreshold:    1.0,
            Stream:            true,
            Question:             question,
            PromptName:        "text",
            ModelName:         setting.Aimodel,
        },
    )
    if err != nil {
        c.JSON(400, gin.H{
            "anwser": "error",
        })
        return
    }
    go func() {
        defer stream.Close()
        defer close(chanStream)
        for {
            var response client.ChatResponse
            response, err = stream.Recv()
            if errors.Is(err, io.EOF) {
                break
            }
            if err != nil {
                fmt.Printf("Stream error: %v\n", err)
                break
            }
            chanStream <- response
        }
    }()
    c.Header("Content-Type", "text/event-stream")
    c.Header("Cache-Control", "no-cache")
    c.Header("Connection", "keep-alive")
    c.Stream(func(w io.Writer) bool {
        if msg, ok := <-chanStream; ok {
            if msg.Answer != "" {
                m := gin.H{
                    "answer":     msg.Answer,
                    "docs":       msg.Docs,
                    "tools":      msg.Tools,
                    "answerstep": msg.AnswerStep,
                }
                jcontent, _ := json.Marshal(m)
                w.Write(jcontent)
            } else {
                m := gin.H{
                    "docs":       msg.Docs,
                    "tools":      msg.Tools,
                    "answerstep": msg.AnswerStep,
                }
                jcontent, _ := json.Marshal(m)
                w.Write(jcontent)
            }
            log.Infof("message: %v\n", msg)
            return true
        }
        return false
    })
}

The response not result as a text/event-stream .It's return all the results at once.

I'm also test using c.SSEvent("message", msg), It's also returned all the results at once.

Actual result

returned all the result at once.

Environment

• go version:1.20
• gin version (or commit ref):1.9.1
• operating system: CentOS 7

[haxung]

Looks like chanStream received response (chanStream <- response in Goroutine) so quickly that like Gin return all results at once. How about add one line time.Sleep(time.Duration(1) * time.Second) after chanStream <- response?