search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.57k stars 4.82k forks source link

[BUG] Updates and publishing QuestionAnswering APIs too slow #30332

Closed tung238 closed 3 months ago

tung238 commented 2 years ago

Library name and version

Azure.AI.Language.QuestionAnswering 1.1.0-beta.2

Describe the bug

We are using Azure QuestionAnswering for our product and recently we saw that all QuestionAnswering's APIs we used to update QnA, Source, and publish Knowledgebase become too slow, previously it usually takes 5 seconds but now takes ~30 seconds to complete. APIs which we notice the slowness in QuestionAnsweringProjectsClient: UpdateSourcesAsync UpdateQnasAsync DeployProjectAsync

This is a sample result of API call:

{{
  'createdDateTime': '2022-08-03T08:31:27+00:00',
  'expirationDateTime': '2022-08-03T14:31:27+00:00',
  'jobId': '9cbe1191-248d-422f-9736-162b6af11d58',
  'lastUpdatedDateTime': '2022-08-03T08:31:35+00:00',
  'status': 'succeeded',
  'resultUrl': '/knowledgebases/TrabbleKnowledgebase'
}}

It looks like azure takes 7 seconds to complete but my side only receives this 30 seconds after trigger request. This happens even with our staging environment on azure so it is not an internet connectivity issue. The issue also happened with Azure.AI.Language.QuestionAnswering library version 1.1.0-beta.1

Time running the benchmark:


BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
11th Gen Intel Core i7-11800H 2.30GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK=6.0.201
  [Host]     : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT
  DefaultJob : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT
Method Mean Error StdDev
UpdateSourcesAsync 30.44 s 0.082 s 0.069 s
UpdateQnasAsync 30.44 s 0.111 s 0.098 s
DeployProjectAsync 30.36 s 0.048 s 0.037 s

Expected behavior

APIs should complete in ~5 seconds

Actual behavior

APIs take ~30 seconds to complete

Reproduction Steps

I created a minimal project for reproducing the issue https://github.com/tung238/QuestionAnsweringIssue

Environment

.NET SDK (reflecting any global.json): Version: 6.0.201 Commit: ef40e6aa06

Runtime Environment: OS Name: Windows OS Version: 10.0.22000 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\6.0.201\

Host (useful for support): Version: 6.0.3 Commit: c24d9a9c91

.NET SDKs installed: 3.1.416 [C:\Program Files\dotnet\sdk] 5.0.405 [C:\Program Files\dotnet\sdk] 6.0.102 [C:\Program Files\dotnet\sdk] 6.0.200 [C:\Program Files\dotnet\sdk] 6.0.201 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

azure-sdk commented 2 years ago

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Storage:0.3783393,Extensions:0.05948581,Tables:0.051196944'

m-redding commented 2 years ago

Thank you for your feedback. Tagging and routing to the team member best able to assist.

heaths commented 2 years ago

@rokulka can you take a look? Our default polling is every 1s if the service doesn't send back a Retry-After header, so any change to timing is not likely the SDK.

ghost commented 2 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @bingisbestest, @nerajput1607.

Issue Details
### Library name and version Azure.AI.Language.QuestionAnswering 1.1.0-beta.2 ### Describe the bug We are using Azure QuestionAnswering for our product and recently we saw that all QuestionAnswering's APIs we used to update QnA, Source, and publish Knowledgebase become too slow, previously it usually takes 5 seconds but now takes ~30 seconds to complete. APIs which we notice the slowness in QuestionAnsweringProjectsClient: UpdateSourcesAsync UpdateQnasAsync DeployProjectAsync This is a sample result of API call: ``` {{ 'createdDateTime': '2022-08-03T08:31:27+00:00', 'expirationDateTime': '2022-08-03T14:31:27+00:00', 'jobId': '9cbe1191-248d-422f-9736-162b6af11d58', 'lastUpdatedDateTime': '2022-08-03T08:31:35+00:00', 'status': 'succeeded', 'resultUrl': '/knowledgebases/TrabbleKnowledgebase' }} ``` It looks like azure takes 7 seconds to complete but my side only receives this 30 seconds after trigger request. This happens even with our staging environment on azure so it is not an internet connectivity issue. The issue also happened with Azure.AI.Language.QuestionAnswering library version 1.1.0-beta.1 Time running the benchmark: ``` ini BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000 11th Gen Intel Core i7-11800H 2.30GHz, 1 CPU, 16 logical and 8 physical cores .NET SDK=6.0.201 [Host] : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT DefaultJob : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT ``` | Method | Mean | Error | StdDev | |------------------- |--------:|--------:|--------:| | UpdateSourcesAsync | 30.44 s | 0.082 s | 0.069 s | | UpdateQnasAsync | 30.44 s | 0.111 s | 0.098 s | | DeployProjectAsync | 30.36 s | 0.048 s | 0.037 s | ### Expected behavior APIs should complete in ~5 seconds ### Actual behavior APIs take ~30 seconds to complete ### Reproduction Steps I created a minimal project for reproducing the issue `https://github.com/tung238/QuestionAnsweringIssue` ### Environment .NET SDK (reflecting any global.json): Version: 6.0.201 Commit: ef40e6aa06 Runtime Environment: OS Name: Windows OS Version: 10.0.22000 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\6.0.201\ Host (useful for support): Version: 6.0.3 Commit: c24d9a9c91 .NET SDKs installed: 3.1.416 [C:\Program Files\dotnet\sdk] 5.0.405 [C:\Program Files\dotnet\sdk] 6.0.102 [C:\Program Files\dotnet\sdk] 6.0.200 [C:\Program Files\dotnet\sdk] 6.0.201 [C:\Program Files\dotnet\sdk] .NET runtimes installed: Microsoft.AspNetCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Author: tung238
Assignees: heaths
Labels: `Service Attention`, `customer-reported`, `question`, `needs-team-attention`, `Service`, `Cognitive - QnA Maker`
Milestone: -
heaths commented 2 years ago

I have verified that the service is sending back a Retry-After: 00:00:30 header, which the newer SDKs honor. @peytonfraser @rokulka this might be something for the service to consider.

jasondotackmanatjenzabardotcom commented 1 year ago

This is a major issue for our products that use QuestionAnswering as well. And ideas if/when this may be resolved? @peytonfraser @rokulka @heaths

github-actions[bot] commented 3 months ago

Hi @tung238, we deeply appreciate your input into this project. Regrettably, this issue has remained unresolved for over 2 years and inactive for 30 days, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.