search

dotnet / aspire

An opinionated, cloud ready stack for building observable, production ready, distributed applications in .NET
https://learn.microsoft.com/dotnet/aspire
MIT License
3.64k stars 411 forks source link

Some tests timing out on helix #5247

Open radical opened 1 month ago

radical commented 1 month ago

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618&view=results Build error leg or test failing: Aspire.Hosting.Elasticsearch.Tests.WorkItemExecution Pull request: https://github.com/dotnet/aspire/pull/5243

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Aborting test run: test run timeout of [0-9]+ milliseconds exceeded",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618 Error message validated: [Aborting test run: test run timeout of [0-9]+ milliseconds exceeded] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 9/11/2024 7:28:44 PM UTC

Report

Build Definition Test Pull Request
810372 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5729
810299 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5538
810218 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5538
810045 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution
809718 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5402
809705 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5621
809615 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5730
809584 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5725
809085 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution
808814 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5690
808485 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5690
808424 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5319
807745 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5538
808417 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5703
807822 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5592
807801 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5688
807605 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5511
807395 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution
807364 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5701
807341 dotnet/aspire Aspire.Hosting.Milvus.Tests.MilvusFunctionalTests.Aspire.Hosting.Milvus.Tests.MilvusFunctionalTests.WithDataShouldPersistStateBetweenUsages dotnet/aspire#5701
807298 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5511
807292 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5699
807319 dotnet/aspire Aspire.Hosting.SqlServer.Tests.SqlServerFunctionalTests.Aspire.Hosting.SqlServer.Tests.SqlServerFunctionalTests.WithDataShouldPersistStateBetweenUsages(useVolume: False)
807228 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5538
807198 dotnet/aspire Aspire.Hosting.Containers.Tests.WithDockerfileTests.Aspire.Hosting.Containers.Tests.WithDockerfileTests.WithBuildSecretPopulatesSecretFilesCorrectly dotnet/aspire#5511
807197 dotnet/aspire Aspire.Hosting.MongoDB.Tests.MongoDbFunctionalTests.Aspire.Hosting.MongoDB.Tests.MongoDbFunctionalTests.VerifyWithInitBindMount dotnet/aspire#5690
807190 dotnet/aspire Aspire.Hosting.MongoDB.Tests.MongoDbFunctionalTests.Aspire.Hosting.MongoDB.Tests.MongoDbFunctionalTests.VerifyWithInitBindMount dotnet/aspire#5592
807177 dotnet/aspire Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos) dotnet/aspire#5697
807139 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5538
807107 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5575
807124 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5695
807077 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5682
807076 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5684
806754 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5694
806452 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5538
806443 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
806042 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5511
805552 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5628
805508 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5505
805406 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5505
805343 dotnet/aspire Aspire.EndToEnd.Tests-cosmos.WorkItemExecution
804618 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5628
804564 dotnet/aspire Aspire.Hosting.MySql.Tests.MySqlFunctionalTests.Aspire.Hosting.MySql.Tests.MySqlFunctionalTests.WithDataShouldPersistStateBetweenUsages dotnet/aspire#5628
804393 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
804474 dotnet/aspire Aspire.Hosting.MySql.Tests.MySqlFunctionalTests.Aspire.Hosting.MySql.Tests.MySqlFunctionalTests.WithDataShouldPersistStateBetweenUsages dotnet/aspire#5628
2535546 dotnet-aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution
804315 dotnet/aspire Aspire.Hosting.MySql.Tests.MySqlFunctionalTests.Aspire.Hosting.MySql.Tests.MySqlFunctionalTests.WithDataShouldPersistStateBetweenUsages dotnet/aspire#5628
803164 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
803149 dotnet/aspire Aspire.Oracle.EntityFrameworkCore.Tests.WorkItemExecution dotnet/aspire#5613
803089 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#5505
803088 dotnet/aspire Aspire.Hosting.Redis.Tests.WorkItemExecution dotnet/aspire#5511
803023 dotnet/aspire Aspire.Hosting.Redis.Tests.WorkItemExecution dotnet/aspire#5511
802103 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution
802093 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5526
802081 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5613
802072 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5611
802082 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5614
802076 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5597
802071 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5610
802070 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5609
802063 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5607
801938 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5604
801925 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5605
801806 dotnet/aspire Aspire.Hosting.Redis.Tests.WorkItemExecution dotnet/aspire#5511
801368 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5578
801366 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5479
800529 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5505
798711 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5553
798248 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
798151 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
798077 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
797988 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
797851 dotnet/aspire Aspire.Workload.Tests.Xunit_PerTestFrameworkTemplatesTests.WorkItemExecution dotnet/aspire#5505
797470 dotnet/aspire Aspire.Hosting.AWS.Tests.WorkItemExecution dotnet/aspire#2225
796984 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution
796961 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5514
796874 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
796692 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
795312 dotnet/aspire Aspire.Hosting.AWS.Tests.WorkItemExecution dotnet/aspire#2225
795145 dotnet/aspire Aspire.Hosting.Redis.Tests.WorkItemExecution dotnet/aspire#5511
794702 dotnet/aspire Aspire.Hosting.AWS.Tests.WorkItemExecution dotnet/aspire#2225
794404 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
794364 dotnet/aspire Aspire.Pomelo.EntityFrameworkCore.MySql.Tests.WorkItemExecution dotnet/aspire#5483
788826 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution
787923 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5426
787983 dotnet/aspire Aspire.Hosting.AWS.Tests.WorkItemExecution dotnet/aspire#2225
787943 dotnet/aspire Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution dotnet/aspire#5425
2523055 dotnet-aspire Aspire.Hosting.Python.Tests.WorkItemExecution
787090 dotnet/aspire Aspire.Hosting.SqlServer.Tests.SqlServerFunctionalTests.Aspire.Hosting.SqlServer.Tests.SqlServerFunctionalTests.VerifySqlServerResource dotnet/aspire#5411
2522952 dotnet-aspire Aspire.Hosting.Python.Tests.WorkItemExecution
781165 dotnet/aspire Aspire.Hosting.SqlServer.Tests.WorkItemExecution dotnet/aspire#5350

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
8 49 91
radical commented 1 month ago

cc @eerhardt @sebastienros

One of the tests timing out is Aspire.Hosting.Elasticsearch.Tests - log.

Why is this timing out after 10minutes on helix when all the tests combined didn't take that long on the build machine?

radical commented 1 month ago

I chose a broader error message to match against, so we can get any tests that are timing out on helix now.

mitchdenny commented 1 month ago

I am still hitting this today on this PR: https://github.com/dotnet/aspire/pull/5223

radical commented 1 month ago

I am still hitting this today on this PR: #5223

I'll bump the timeouts. We are hitting new ones as we moved new tests to helix last week.

radical commented 1 month ago

Aspire.Hosting.Elasticsearch.Tests - I think this is timing out because each of the functional tests starts a new app, and the elasticsearch containers take a few minutes to start up, causing the time taken for all the tests to be more than 10 minutes.