search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.37k stars 4.78k forks source link

[Synapse] TestSparkBatchJobCompletesWhenJobStarts failing in nightly runs #31935

Open AFengKK opened 2 years ago

AFengKK commented 2 years ago

Synapse nightly test runs are failing with:

Error message Azure.RequestFailedException : Service request failed. Status: 401 (Unauthorized)

Content: {"code":"AuthenticationFailed","message":"Token Authentication failed - IDX10222: Lifetime validation failed. The token is not yet valid. ValidFrom: '1/20/2023 4:08:49 AM', Current time: '10/18/2022 7:16:35 PM'."}

Headers: Server: Microsoft-HTTPAPI/2.0 Strict-Transport-Security: REDACTED x-ms-request-id: a36e8040-6804-474f-8392-e263e4adbb24 x-ms-client-request-id: fc979fa4-e412-44c0-b867-0114ec717680 Date: Tue, 18 Oct 2022 19:16:35 GMT Content-Length: 213 Content-Type: application/json; charset=utf-8

Stack trace at Azure.Analytics.Synapse.Spark.SparkBatchRestClient.CreateSparkBatchJob(SparkBatchJobOptions sparkBatchJobOptions, Nullable1 detailed, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/sdk/synapse/Azure.Analytics.Synapse.Spark/src/Generated/SparkBatchRestClient.cs:line 203 at Azure.Analytics.Synapse.Spark.SparkBatchClient.StartCreateSparkBatchJobInternal(Boolean async, SparkBatchJobOptions sparkBatchJobOptions, Nullable1 detailed, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/sdk/synapse/Azure.Analytics.Synapse.Spark/src/Customization/SparkBatchClient.cs:line 36 at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](Task1 task) in /mnt/vss/_work/1/s/sdk/core/Azure.Core/src/Shared/TaskExtensions.cs:line 33 at Azure.Analytics.Synapse.Spark.SparkBatchClient.StartCreateSparkBatchJob(SparkBatchJobOptions sparkBatchJobOptions, Nullable1 detailed, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/sdk/synapse/Azure.Analytics.Synapse.Spark/src/Customization/SparkBatchClient.cs:line 21 --- End of stack trace from previous location --- at Azure.Core.TestFramework.DiagnosticScopeValidatingInterceptor.<>cDisplayClass4_01.<<WrapAsyncResultCore>g__Await|0>d.MoveNext() in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/DiagnosticScopeValidatingInterceptor.cs:line 120 --- End of stack trace from previous location --- at Azure.Core.TestFramework.InstrumentResultInterceptor.InstrumentOperationInterceptor[T](IInvocation invocation, Func1 innerTask) in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/InstrumentResultInterceptor.cs:line 105 at Azure.Core.TestFramework.DiagnosticScopeValidatingInterceptor.<>c__DisplayClass4_0`1.<gAwait|0>d.MoveNext() in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/DiagnosticScopeValidatingInterceptor.cs:line 120 --- End of stack trace from previous location --- at Azure.Core.TestFramework.DiagnosticScopeValidatingInterceptor.<>cDisplayClass5_01.<<ValidateDiagnosticScopeInterceptor>b__0>d.MoveNext() in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/DiagnosticScopeValidatingInterceptor.cs:line 171 --- End of stack trace from previous location --- at Azure.Core.TestFramework.DiagnosticScopeValidatingInterceptor.ValidateDiagnosticScope[T](Func1 action, MethodInfo methodInfo, String source) in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/DiagnosticScopeValidatingInterceptor.cs:line 201 at Azure.Analytics.Synapse.Spark.Tests.SparkBatchClientLiveTests.TestSparkBatchJobCompletesWhenJobStarts() in /mnt/vss/_work/1/s/sdk/synapse/Azure.Analytics.Synapse.Spark/tests/SparkBatchClientLiveTests.cs:line 46 at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter1.GetResult() at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func1 invoke) at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context) at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context) at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>cDisplayClass1_0.b__0() at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)

For more details check here:

@jsquire for notification.

azure-sdk commented 2 years ago

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Azure.Core:0.5947542,Synapse:0.17048396,AutoRest Runtime:0.061088532'

ghost commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @wonner, @yanjungao718.

Issue Details
Synapse nightly test runs are failing with: > Error message > Azure.RequestFailedException : Service request failed. > Status: 401 (Unauthorized) > > Content: > {"code":"AuthenticationFailed","message":"Token Authentication failed - IDX10222: Lifetime validation failed. The token is not yet valid. ValidFrom: '1/20/2023 4:08:49 AM', Current time: '10/18/2022 7:16:35 PM'."} > > Headers: > Server: Microsoft-HTTPAPI/2.0 > Strict-Transport-Security: REDACTED > x-ms-request-id: a36e8040-6804-474f-8392-e263e4adbb24 > x-ms-client-request-id: fc979fa4-e412-44c0-b867-0114ec717680 > Date: Tue, 18 Oct 2022 19:16:35 GMT > Content-Length: 213 > Content-Type: application/json; charset=utf-8 > > > Stack trace > at Azure.Analytics.Synapse.Spark.SparkBatchRestClient.CreateSparkBatchJob(SparkBatchJobOptions sparkBatchJobOptions, Nullable`1 detailed, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/sdk/synapse/Azure.Analytics.Synapse.Spark/src/Generated/SparkBatchRestClient.cs:line 203 > at Azure.Analytics.Synapse.Spark.SparkBatchClient.StartCreateSparkBatchJobInternal(Boolean async, SparkBatchJobOptions sparkBatchJobOptions, Nullable`1 detailed, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/sdk/synapse/Azure.Analytics.Synapse.Spark/src/Customization/SparkBatchClient.cs:line 36 > at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](Task`1 task) in /mnt/vss/_work/1/s/sdk/core/Azure.Core/src/Shared/TaskExtensions.cs:line 33 > at Azure.Analytics.Synapse.Spark.SparkBatchClient.StartCreateSparkBatchJob(SparkBatchJobOptions sparkBatchJobOptions, Nullable`1 detailed, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/sdk/synapse/Azure.Analytics.Synapse.Spark/src/Customization/SparkBatchClient.cs:line 21 > --- End of stack trace from previous location --- > at Azure.Core.TestFramework.DiagnosticScopeValidatingInterceptor.<>c__DisplayClass4_0`1.<g__Await|0>d.MoveNext() in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/DiagnosticScopeValidatingInterceptor.cs:line 120 > --- End of stack trace from previous location --- > at Azure.Core.TestFramework.InstrumentResultInterceptor.InstrumentOperationInterceptor[T](IInvocation invocation, Func`1 innerTask) in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/InstrumentResultInterceptor.cs:line 105 > at Azure.Core.TestFramework.DiagnosticScopeValidatingInterceptor.<>c__DisplayClass4_0`1.<g__Await|0>d.MoveNext() in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/DiagnosticScopeValidatingInterceptor.cs:line 120 > --- End of stack trace from previous location --- > at Azure.Core.TestFramework.DiagnosticScopeValidatingInterceptor.<>c__DisplayClass5_0`1.<b__0>d.MoveNext() in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/DiagnosticScopeValidatingInterceptor.cs:line 171 > --- End of stack trace from previous location --- > at Azure.Core.TestFramework.DiagnosticScopeValidatingInterceptor.ValidateDiagnosticScope[T](Func`1 action, MethodInfo methodInfo, String source) in /mnt/vss/_work/1/s/sdk/core/Azure.Core.TestFramework/src/Instrumentation/DiagnosticScopeValidatingInterceptor.cs:line 201 > at Azure.Analytics.Synapse.Spark.Tests.SparkBatchClientLiveTests.TestSparkBatchJobCompletesWhenJobStarts() in /mnt/vss/_work/1/s/sdk/synapse/Azure.Analytics.Synapse.Spark/tests/SparkBatchClientLiveTests.cs:line 46 > at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.GetResult() > at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke) > at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context) > at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context) > at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.b__0() > at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action) For more details check here: - https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1927024&view=results @jsquire for notification.
Author: AFengKK
Assignees: -
Labels: `Service Attention`, `Client`, `Synapse`, `test-reliability`
Milestone: -