Add retry for AddressInUseException during aspnetcore tests

[akoeplinger]

We're seeing occasional test failures in the VMR:

All of them are a variation of the below, due to the port being already used:

    Test environment:
      Dotnet Root: /Users/runner/work/1/s/artifacts/obj/extracted-dotnet-sdk/
      Test root: /Users/runner/work/1/s/artifacts/scenario-tests/artifacts/
      Target RID: osx-arm64
      Sdk Version: 9.0.100-preview.5.24273.1
      Platform: OSX
    [FAIL] Microsoft.DotNet.ScenarioTests.SdkTemplateTests.SdkTemplateTests.VerifyWebAPITemplate(language: CSharp)
    System.InvalidOperationException : Failed to execute /Users/runner/work/1/s/artifacts/obj/extracted-dotnet-sdk/dotnet run /bl:/Users/runner/work/1/s/artifacts/scenario-tests/artifacts/SdkTemplateTest_WebApi_CSharp/run.binlog
    Exit code: 134
    Building...
    fail: Microsoft.Extensions.Hosting.Internal.Host[11]
          Hosting failed to start
          System.IO.IOException: Failed to bind to address http://127.0.0.1:5295/: address already in use.
           ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
           ---> System.Net.Sockets.SocketException (48): Address already in use
             at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
             at System.Net.Sockets.Socket.Bind(EndPoint localEP)
             at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
             at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
             --- End of inner exception stack trace ---
             at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
             at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
             at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
             at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()

We should add some sort of retry.

Note that the way the templates work is that dotnet new generates a random port and writes it into Properties/launchSettings.json so just rerunning the app will hit the same port, we'd presumably need to delete and recreate the project.

/cc @ViktorHofer @mthalman

[ViktorHofer]

Can we override the port specified in launchSettings.json at startup?

[akoeplinger]

Yeah we can e.g. set ASPNETCORE_URLS=http://localhost:8080 env var to override that, but I'm not sure whether that could hide issues.

[ViktorHofer]

Interesting. What about a hill climbing wait and make the test app wait until the port is free again? Given that the number of ports is limited this might be needed anyway. I don't know how many ports are opened in total by aspnetcore scenario tests.

[akoeplinger]

What about a hill climbing wait and make the test app wait until the port is free again?

I don't think that'd work since I've seen at least one failure where we hit a port that is used by some system service so that one would never be free.

I don't know how many ports are opened in total by aspnetcore scenario tests.

Right now we only run a handful of tests so we could also define our own pool of port numbers in a safe range and let tests grab from there while not reusing ports. I'm just not sure we should override this as it might impact the validity of the scenario tests.