[dev-tool] test-proxy wrapper doesn't handle parallel execution

[jeremymeng]

Hit this error while exploring mocha upgrade.

To repro: I have a WIP branch here https://github.com/jeremymeng/azure-sdk-for-js/tree/engsys/mocha-v10. just need to rush update, rush build, rush unit-test:node

Alternatively, maybe manually running playback tests for two packages at the same time

[19:10:39] info: Microsoft.Hosting.Lifetime[0] Content root path: /home/meng/git/jsmain/sdk/core/abort-controller [02:10:39] Recorded: 0 Played Back: 0 [02:10:40] Recorded: 0 Played Back: 0 Running proxy version is Azure.Sdk.Tools.TestProxy 20230818.1 git --version [20:21:48] info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[62] User profile is available. Using '/home/meng/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest. Unhandled exception: System.IO.IOException: Failed to bind to address http://127.0.0.1:5000: address already in use. ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use ---> System.Net.Sockets.SocketException (98): Address already in use at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName) at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.Sockets.Socket.Bind(EndPoint localEP) at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint) at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind() --- End of inner exception stack trace --- at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind() at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>cDisplayClass30_01.<<StartAsync>g__OnBind|0>d.MoveNext() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken) --- End of inner exception stack trace --- at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.DefaultAddressStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IEnumerable1 listenOptions, AddressBindContext context, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken) at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host) at Azure.Sdk.Tools.TestProxy.Startup.StartServer(StartOptions startOptions) in /mnt/vss/_work/1/s/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Startup.cs:line 181 at Azure.Sdk.Tools.TestProxy.Startup.Run(Object commandObj) in /mnt/vss/_work/1/s/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Startup.cs:line 100 at Azure.Sdk.Tools.TestProxy.CommandOptions.OptionsGenerator.<>c__DisplayClass0_0.<b6>d.MoveNext() in /mnt/vss/_work/1/s/tools/test-proxy/Azure.Sdk.Tools.TestProxy/CommandOptions/OptionsGenerator.cs:line 80 --- End of stack trace from previous location --- at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context) at System.CommandLine.Invocation.InvocationPipeline.<>cDisplayClass4_0.<b0>d.MoveNext() --- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>cDisplayClass17_0.<b0>d.MoveNext() --- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>cDisplayClass12_0.<b0>d.MoveNext() --- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>cDisplayClass22_0.<b0>d.MoveNext() --- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<b__0>d.MoveNext() --- End of stack trace from previous location --- Running proxy version is Azure.Sdk.Tools.TestProxy 20230818.1

[jeremymeng]

Maybe just check process list before spawning? Assuming test-proxy can serve multiple packages concurrently. @timovv @HarshaNalluru

[HarshaNalluru]

Should be able to support multiple packages concurrently with one single run of test proxy. I will take a look and reach out to you.

[HarshaNalluru]

I have app-config(right) and storage-blob(left) packages running tests in playback in parallel, and I do not repro the issue.

Branch: jeremymeng/azure-sdk-for-js at engsys/mocha-v10 (github.com)

[jeremymeng]

we already check whether test-proxy is active or not by making a request to it. This could be a race condition when the tool is starting but not ready yet, another process tries to start too

[HarshaNalluru]

Interesting, will investigate more.

[jeremymeng]

Another problem: even if we could detect that other package's test script started test proxy, the other package's test could end and test proxy stopped, thus causing failure. It feels that we need some way to ref count clients and stop server only when there's no active clients but this could make the test proxy to appear "hanging"? Another alternative is to start multiple instance of server each with its own unique port?