[wasm] WBT: Blazor tests on windows failing with `Failed to bind to address` for random ports

[radical]

Reproduction Steps

• Queue: browser-wasm linux Release WasmBuildTests
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=413445&view=results
• Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-heads-release-80-rc1-e5e75e14b32149239e/NoWorkload-Wasm.Build.Tests/1/console.cabf329c.log?helixlogtype=result
• Output:

  
  Wasm.Build.Tests.Blazor.WorkloadRequiredTests.WorkloadNotRequiredForInvariantGlobalization(config: "Release", invariant: False, publish: False) [FAIL]
     Expected 0 exit code but got 134: /root/helix/work/workitem/e/dotnet-none/dotnet run -c Release --no-build
    Standard Output:
    [] fail: Microsoft.Extensions.Hosting.Internal.Host[11]
    []       Hosting failed to start
    []       System.IO.IOException: Failed to bind to address http://127.0.0.1:5145: address already in use.
    []        ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
    []        ---> System.Net.Sockets.SocketException (98): Address already in use
    []          at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, Boolean disconnectOnFailure, String callerName)
    []          at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
    []          at System.Net.Sockets.Socket.Bind(EndPoint localEP)
    []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
    []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
    []          --- End of inner exception stack trace ---
    []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
    []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
    []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
    []          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
    []       --- End of stack trace from previous location ---
    []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
    []          --- End of inner exception stack trace ---
    []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
    []          at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
    []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
    []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(ListenOptions[] listenOptions, AddressBindContext context, Func`2 useHttps, CancellationToken cancellationToken)
    []          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
    []          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
    []          at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
    []          at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
    []          at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
    [] Unhandled exception. System.IO.IOException: Failed to bind to address http://127.0.0.1:5145: address already in use.
    []  ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
    []  ---> System.Net.Sockets.SocketException (98): Address already in use
    []    at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, Boolean disconnectOnFailure, String callerName)
    []    at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
    []    at System.Net.Sockets.Socket.Bind(EndPoint localEP)
    []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
    []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
    []    --- End of inner exception stack trace ---
    []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
    []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
    []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
    []    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
    [] --- End of stack trace from previous location ---
    []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
    []    --- End of inner exception stack trace ---
    []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
    []    at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
    []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
    []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(ListenOptions[] listenOptions, AddressBindContext context, Func`2 useHttps, CancellationToken cancellationToken)
    []    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
    []    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
    []    at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
    []    at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
    []    at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
    []    at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
    []    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
    []    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
    []    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
    []    at Microsoft.AspNetCore.Components.WebAssembly.DevServer.Program.Main(String[] args)

<!-- Error message template  -->
### Known Issue Error Message

Fill the error message using [step by step known issues guidance](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md#how-to-fill-out-a-known-issue-error-section).

<!-- Use ErrorMessage for String.Contains matches. Use ErrorPattern for regex matches (single line/no backtracking). Set BuildRetry to `true` to retry builds with this error. Set ExcludeConsoleLog to `true` to skip helix logs analysis. -->

```json
{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "Microsoft.AspNetCore.Connections.AddressInUseException:",
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: Result validation: :warning: Build internal information not found. This may happen if your build is too old. Please use a build that is no older than two weeks. If the problem persists, contact .NET Engineering Services Team and share this issue. Validation performed at: 10/17/2023 1:46:13 AM UTC

Report

Build	Definition	Test	Pull Request
676400	dotnet/runtime	Wasm.Build.Tests.Blazor.BuildPublishTests.DefaultTemplate_WithoutWorkload	dotnet/runtime#102187
671211	dotnet/runtime	Wasm.Build.Tests.Blazor.BuildPublishTests.DefaultTemplate_WithoutWorkload	dotnet/runtime#102064
662830	dotnet/runtime	Wasm.Build.Tests.Blazor.WorkloadRequiredTests.WorkloadNotRequiredForInvariantGlobalization
661214	dotnet/runtime	Wasm.Build.Tests.Blazor.WorkloadRequiredTests.WorkloadNotRequiredForInvariantGlobalization	dotnet/runtime#101277
656049	dotnet/runtime	Wasm.Build.Tests.Blazor.BuildPublishTests.DefaultTemplate_WithoutWorkload	dotnet/runtime#100141

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	5

[ghost]

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Error Blob Tagging this for KnownBuildError in case it's happening consistently and we need to fix this in 8.0. ```json { "ErrorMessage": "", "BuildRetry": false, "ErrorPattern": "Wasm\.Build\.Tests.*System\.IO\.IOException: Failed to bind to address http://127\.0\.0\.1:(?!5000)", "ExcludeConsoleLog": false } ``` ### Reproduction Steps - Queue: `browser-wasm linux Release WasmBuildTests` - Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=413445&view=results - Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-heads-release-80-rc1-e5e75e14b32149239e/NoWorkload-Wasm.Build.Tests/1/console.cabf329c.log?helixlogtype=result ### Known Issue Error Message Fill the error message using [step by step known issues guidance](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md#how-to-fill-out-a-known-issue-error-section). ```json { "ErrorMessage": "", "ErrorPattern": "", "BuildRetry": false, "ExcludeConsoleLog": false } ```

Author:	radical
Assignees:	-
Labels:	`arch-wasm`, `blocking-clean-ci`, `Known Build Error`
Milestone:	-

[radical]

@missymessa Any idea why this is not getting picked up for known-build-error analysis?

[radical]

@maraf Looking at the logs, the same port number is used in one of the earlier tests, but even if that process is still running the devserver shouldn't have picked a port still in use. Seems like a devserver issue?

[AlitzelMendez]

@radical it wasn't picked up because it had escaping problems, you can notice this because the json message was in red, like this:

{
 "ErrorPattern": "Wasm\.Build\.Tests.*System\.IO\.IOException: Failed to bind to address http://127\.0\.0\.1:(?!5000)"
}

I fixed the escaping and now the validation was able to run, but I noticed that your regex was trying to look into multiple lines, this feature is not available for known issues, the error needs to be a single line, also the validation returned a backtracking error, all this needs to be fixed in order than known issues can work

let me know if you need help with any of this

[radical]

I fixed the escaping and now the validation was able to run, but I noticed that your regex was trying to look into multiple lines, this feature is not available for known issues,

Oh, I was basing this on the doc:

https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md#regex-matching

We recommend you test your regular expression, to do it you can use [regex101 tester](https://regex101.com/) (choose .NET (C#) flavor) with the following regex options:

Single line
Insentitive
No backtracking

The single line here would mean that . would match newlines also, even according to that website.

If that isn't supported then what is the recommended way to identify failure messages that are not unique. For example System.IO.IOException: Failed to bind to address can match lots of test failures but the issue is specifically about this failure in Wasm.Build.Tests. Without the multi-line regex, how do I narrow down the match?

[AlitzelMendez]

If that isn't supported then what is the recommended way to identify failure messages that are not unique. For example System.IO.IOException: Failed to bind to address can match lots of test failures but the issue is specifically about this failure in Wasm.Build.Tests. Without the multi-line regex, how do I narrow down the match?

Unfortunately, there is not a lot that we can suggest on these cases, if this is narrow to a specific test name you can use the test name, otherwise the only option is try to identify an error line that is unique to your issue.

we provide a little bit more information about why the multiline is not supported of this issue: https://github.com/dotnet/arcade/issues/13253

[radical]

If that isn't supported then what is the recommended way to identify failure messages that are not unique. For example System.IO.IOException: Failed to bind to address can match lots of test failures but the issue is specifically about this failure in Wasm.Build.Tests. Without the multi-line regex, how do I narrow down the match?

Unfortunately, there is not a lot that we can suggest on these cases, if this is narrow to a specific test name you can use the test name, otherwise the only option is try to identify an error line that is unique to your issue.

we provide a little bit more information about why the multiline is not supported of this issue: dotnet/arcade#13253

Maybe allowing matching against the helix item name, or maybe two single line patterns to match? That should be cheaper than a multiline regex search over a big text file.

[pavelsavara]

does it not happen anymore ? Or the filter doesn't work ?