search

dotnet / aspire

Tools, templates, and packages to accelerate building observable, production-ready apps
https://learn.microsoft.com/dotnet/aspire
MIT License
3.92k stars 479 forks source link

Some tests timing out on helix #5247

Open radical opened 3 months ago

radical commented 3 months ago

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618&view=results Build error leg or test failing: Aspire.Hosting.Elasticsearch.Tests.WorkItemExecution Pull request: https://github.com/dotnet/aspire/pull/5243

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Aborting test run: test run timeout of [0-9]+ milliseconds exceeded",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618 Error message validated: [Aborting test run: test run timeout of [0-9]+ milliseconds exceeded] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 9/11/2024 7:28:44 PM UTC

Report

Build Definition Test Pull Request
875406 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#6682
869551 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#6666
867188 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution
863093 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution
861137 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#6610
859009 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#6589
857247 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#6570
856981 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#6057
856858 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#6560
856341 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#6546
856290 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
854328 dotnet/aspire Aspire.Playground.Tests.WorkItemExecution dotnet/aspire#6390

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 1 12
radical commented 3 months ago

cc @eerhardt @sebastienros

One of the tests timing out is Aspire.Hosting.Elasticsearch.Tests - log.

Why is this timing out after 10minutes on helix when all the tests combined didn't take that long on the build machine?

radical commented 3 months ago

I chose a broader error message to match against, so we can get any tests that are timing out on helix now.

mitchdenny commented 3 months ago

I am still hitting this today on this PR: https://github.com/dotnet/aspire/pull/5223

radical commented 3 months ago

I am still hitting this today on this PR: #5223

I'll bump the timeouts. We are hitting new ones as we moved new tests to helix last week.

radical commented 3 months ago

Aspire.Hosting.Elasticsearch.Tests - I think this is timing out because each of the functional tests starts a new app, and the elasticsearch containers take a few minutes to start up, causing the time taken for all the tests to be more than 10 minutes.

mitchdenny commented 3 days ago

This just happened for mongo in the playground tests. I was briefly able to repro a timeout locally as well, but when I stopped the test and retried it worked fine (multiple times). When it was timing out I noticed that mongo express didn't have an external port allocated. Make me think that Docker failed to forward the port into the container. This would explain it waiting forever to be able to connect to the container via the endpoint.