How to stop generation midway with CancellationTokenSource ?

[Sushaokun]

Service

Vllm

Describe the bug

When communicating with the Vllm service, there is no way to stop the build midway like with python.

Steps to reproduce

When I execute stream.response.close() in Python code, vllm will output "Aborted request chat-xxxx.". In C#, when I call cancelSource.Cancel(), the streaming thread in C# does stop. However, the vllm server will continue to generate until all inferences are completed and end with the output "Finished request chat-xxxx".

Code snippets

BackgroundWorker bw = new BackgroundWorker();
 CancellationTokenSource cancelSource;
 public TestForm()
 {
     InitializeComponent();

     bw.WorkerReportsProgress = true;
     bw.WorkerSupportsCancellation = true;
     bw.DoWork += (sender, e) =>
     {
         OpenAIClientOptions option = new OpenAIClientOptions();
         option.Endpoint = new Uri("http://172.18.130.1:8000/v1");
         OpenAIClient client = new("", option);

         cancelSource = new CancellationTokenSource();

         ChatClient cc = client.GetChatClient("chatllm");

         CollectionResult<StreamingChatCompletionUpdate> updates = cc.CompleteChatStreaming([
             new SystemChatMessage(sysp),
             new UserChatMessage(userp)
             ]
             ,
             cancellationToken: cancelSource.Token);

         foreach (StreamingChatCompletionUpdate update in updates)
         {
             foreach (ChatMessageContentPart updatePart in update.ContentUpdate)
             {
                 bw.ReportProgress(0, updatePart.Text);
             }
         }
     };

     bw.ProgressChanged += (sender, e) =>
     {
         richTextBox1.AppendText(e.UserState.ToString());
     };

     bw.RunWorkerCompleted += (sender, e) =>
     {
         button1.Enabled = true;
     };
 }

 private void button1_Click(object sender, EventArgs e)
 {
     richTextBox1.Clear();
     bw.RunWorkerAsync();
     button1.Enabled = false;
 }

 private void button3_Click(object sender, EventArgs e)
 {
     if (bw.IsBusy)
     {
         cancelSource.Cancel();
     }
 }

OS

Windwos 11

.NET version

8.0

Library version

Prepare 2.0.0-beta.

[joseharriaga]

Thank you for reaching out, @Sushaokun ! There was a bug causing CancellationToken to not work appropriately in streaming chat completions which is fixed as of version 2.0.0-beta.13.

[Sushaokun]

Thank you for reaching out, @Sushaokun ! There was a bug causing CancellationToken to not work appropriately in streaming chat completions which is fixed as of version 2.0.0-beta.13.

I downloaded the latest source code (version 2.0.0-beta.13.) and tested it, but the problem still exists.