Address "System.Net.Sockets.SocketException: Address already in use" on K8S/Linux using HttpClient/TCP

[yuezengms]

~Assumption: Duplicate of dotnet/runtime#27274 which was fixed by dotnet/corefx#32046 - goal: Port it (once confirmed it is truly duplicate).~ This is HttpClient/TCP spin off. UdpClient is covered fully by dotnet/runtime#27274.

Issue Title

"System.Net.Sockets.SocketException: Address already in use" on Linux

General

Our .net core(v 2.2.0) services are running on Azure Kubernettes Linux environment. Recently we experimenced a lot of error "System.Net.Http.HttpRequestException: Address already in use" while calling dependencies, e.g. Active Directory, CosmosDB and other services. Once the issue started, we kept getting the same errors and had to restart the service to get rid of it. Our http clients are using DNS address, not specific ip and port. The following is the call stack on one example. What can cause such issues and how to fix it?

System.Net.Http.HttpRequestException: Address already in use ---> 
System.Net.Sockets.SocketException: Address already in use#N#   
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)#N#   --- 
End of inner exception stack trace ---#N#   

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)#N#   
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)#N#   
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)#N#   
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)#N#   
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)#N#   
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)#N#   
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)#N#   
at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.HttpClientWrapper.GetResponseAsync()#N#   
at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T](Boolean respondToDeviceAuthChallenge)#N#   
at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T]()#N#   
at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.SendHttpMessageAsync(IRequestParameters requestParameters)#N#   
at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.SendTokenRequestAsync()#N#   
at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.CheckAndAcquireTokenUsingBrokerAsync()#N#   
at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.RunAsync()#N#   
at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenForClientCommonAsync(String resource, ClientKey clientKey)#N#   
at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenAsync(String resource, ClientCredential clientCredential)#N#   

[stephentoub]

according to the docs, you should not be using HttpClient directly

It's fine to use HttpClient directly. HttpClientFactory layers on top of that to provide additional management. When you do use HttpClient directly, you should reuse instances as much as possible.

[antonioortizpola]

@stephentoub I do not know how to define "fine", and how much is "as much as possible", on the other hand, no, you should not simply hold the client as long as possible, since it will not respect DNS changes.

Again, from the docs

...But there’s a second issue with HttpClient that you can have when you use it as singleton or static object. In this case, a singleton or static HttpClient doesn't respect DNS changes

This is a real problem when you have infrastructure on the cloud, it is not as simple as "hold to your client", that exactly what HttpClientFactory is trying to mitigate

[stephentoub]

In this case, a singleton or static HttpClient doesn't respect DNS changes

This is a real problem when you have infrastructure on the cloud

That information is out-of-date and no longer accurate.

[antonioortizpola]

@stephentoub well, if the docs are wrong then I am lost.

The last update is from 01/06/2019, should I ask for an update?, could you please tell us exactly what is wrong so they can make the updates?

Also, if the best solution is to keep the HttpClient for as long as you can, I should not be better just use it as singleton? this would render the IHttpClientFactory pretty much useless, it would be just a fancy name for singleton.

Also it would be great to make that clear in the Core documentation, that you can use HttpClient as singleton should be an option in the "Making Http requests" part, since it still states:

Manages the pooling and lifetime of underlying HttpClientMessageHandler instances to avoid common DNS problems that occur when manually managing HttpClient lifetimes

[stephentoub]

could you please tell us exactly what is wrong so they can make the updates?

SocketsHttpHandler, which is the default handler implementation backing the HttpClient starting in .NET Core 2.1, has two properties on it: PooledConnectionIdleTimeout (https://docs.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler.pooledconnectionidletimeout?view=netcore-2.2) and PooledConnectionLifetime (https://docs.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler.pooledconnectionlifetime?view=netcore-2.2). The latter governs how long a connection is allowed to be reused. It defaults to Infinite, which means it won't be proactively torn down by the client for reasons of how long its been around. But if you set it to something shorter, like 5 minutes, that tells the handler that once the connection has been around for 5 minutes, it shouldn't be reused again and the handler will prevent further requests from using it and will tear it down. Any subsequent requests would be forced to get a new connection, which will again consult DNS to determine to where the connection should be opened.

[rbrugnollo]

SocketsHttpHandler, which is the default handler implementation backing the HttpClient starting in .NET Core 2.1, has two properties on it

So, nothing was changed from 2.1 to 2.2 that would explain the errors showing up on 2.2 but not 2.1 right?!

[karelz]

So, nothing was changed from 2.1 to 2.2 that would explain the errors showing up on 2.2 but not 2.1 right?!

Correct. We did just handful of targeted servicing-level bug fixes in 2.2/2.2.x over 2.1.x. It is quite possible that the "regression" is caused by another component, or we're just "lucky" to hit it on 2.2 due to "random" reasons. Just to confirm: Did anyone hit it in 2.1 at all?

[antonioortizpola]

Just to confirm: Did anyone hit it in 2.1 at all?

Not for us, I can confirm, no problems in 3 months with 2.1, the problem started the day we switched to 2.2.

I am in the process of make my WSDL clients singletons, I hope to end by the next week, that way I could confirm if it is a problem with the HttpClient alone or if WCF is making something wrong

[rbrugnollo]

Just to confirm: Did anyone hit it in 2.1 at all?

App working for 6 months on 2.1 without any issues, now happening on 2.2.

I'm trying to filter exactly which call is throwing the error, so I can isolate and run more tests.

[karelz]

Based on the replies here, I don't think it will be simple to create a repro (although it would be most helpful). I would recommend to get any repro environment where we can experiment - collect more data, try private builds, etc. If anyone has such environment (incl. production) where they can experiment and work closely with us, please let me know and let's dig deeper into it ...

[simonziegler]

I have set up a number of ASP.NET Core 3.0 trial projects and apply Kubernetes orchestration support. In every single case, the first time I run a debug session everything works fine. Then when I stop the session and start up again I get this error. The only way to get around it is to close VS2019 (not preview version) and restart for the next debugging session.

This does not happen with ASP.NET Core 2.2.

A screen grab of the issue:

[antonioortizpola]

@simonziegler that problem is not related to this issue, please feel free to open a new issue so It can be disused appropriately.

Also, it seems an odd error, like the debuger is not finalizing and releasing the port, I would try to install the latest version of VS and Net Core 3 to be sure, and I would use the "report problem" option shipped within Visual Studio instead of this git which is code related only.

[OpenSourceAries]

System.Net.Http.HttpRequestException: Address already in use ---> System.Net.Sockets.SocketException: Address already in use at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

This exception happens both in .NET Core 2.1(sdk:2.1.505&runtime:2.1.9) and .NET Core 2.2(sdk:2.2.105&runtime:2.2.3) with k8s Enviornment(v1.6.7/v1.9.7) after a long running(few days) and never happens in .NET Core 2.0. I'll refactor each call instance HttpClient with HttpClientFactory to try resolve this problem although this exception may still happen based on the previous reply.

[LukePulverenti]

@LukePulverenti did you validate your problem is indeed the same root cause and is fixed in .NET Core 3.0? (and that it is not just same symptom) There seems to be enough +1s to justify backport, we just need to be sure it is the right fix ... first step would be to validate on 3.0. Then we can cherry pick and ask for private validation on 2.2/2.1 build.

For us, this is the one that we need: https://github.com/dotnet/corefx/pull/32046/files

[karelz]

@LukePulverenti did you confirm that particular change helps your case? Or did you use latest .NET Core 3.0 to validate that? @antonioortizpola above mentioned that the change (in .NET Core 3.0) does not help their scenario at all: https://github.com/dotnet/corefx/issues/37044#issuecomment-486335084

[karelz]

Reopening to track solution at least in 3.0

[karelz]

We still need someone to help us track this down: Anyone has an environment where it happens on somewhat regular basis, where we could work with you to collect more logs and experiment? It would be great help. Thanks!

[wfurt]

It seems like we mixing multiple issues here. Part of the discussion is about UDP and part about HttpClient.

[antonioortizpola]

My problem is with HttpClient, my project has two ways to use it:

• Directly for our restful endpoints
• Indirectly using WCF for soap services

It is the only thing that my project does and we are hitting the issue as this comment states.

Sadly for time pressure we just set a workaround to restart the program each time this happens, and been working in other things, but if it can help, I could separate my calls in two projects so I can be sure if the problem comes from the SOAP services or our restful services

[wfurt]

ok, for HTTP: The error was really puzzling to me. Even of man page to connect() mentions EADDRINUSE, I could not find it while looking at Linux kernel sources.
Only one place I could find, we bind() and we don't use that in HttpClient. It turend out we actually do at Socket.ConnectAsync():

   if (_rightEndPoint == null)
    {
        if (endPointSnapshot.AddressFamily == AddressFamily.InterNetwork)
       {
            InternalBind(new IPEndPoint(IPAddress.Any, 0));
        }
        else if (endPointSnapshot.AddressFamily != AddressFamily.Unix)
        {
            InternalBind(new IPEndPoint(IPAddress.IPv6Any, 0));
        }
    }

and than https://github.com/torvalds/linux/blob/master/net/ipv4/af_inet.c#L526-L532

if (snum || !(inet->bind_address_no_port ||
              force_bind_address_no_port)) {
        if (sk->sk_prot->get_port(sk, snum)) {
            inet->inet_saddr = inet->inet_rcv_saddr = 0;
            err = -EADDRINUSE;
            goto out_release_sock;
        }

So this error can pop up if system runs out of port numbers.

This can also happen if IPAddress.IPv6Any is not available be we try to connect on IPV6/dual-mode socket. But that should be be pretty deterministic and I would not expect it to fail only some times (or be fixed by restart)

I would suggest to check that and for example follow https://www.cyberciti.biz/tips/linux-increase-outgoing-network-sockets-range.html

Note that if anybody can give it try, you can do:

strace -f -o trace.txt -s 200 -t -e trace=connect,bind ./myCoolApp

I know the message is confusing but the bottom of this may be system running out of resources. Also note, that it may be worth of checking process limits for file descriptors and buffers.

It may be worth of monitoring /proc/<PID>/fd to see, if descriptor count is going up.

[tmds]

So this error can pop up if system runs out of port numbers.

This matches with my earlier comment https://github.com/dotnet/corefx/issues/37044#issuecomment-485055490

So either the system is running out of ports due to a limited port range, or HttpClient is leaking sockets (or keeping them open too long).

[softworkz]

Repro Szenario for UDP Bug

@karelz - you wrote:

@LukePulverenti did you validate your problem is indeed the same root cause and is fixed in .NET Core 3.0? (and that it is not just same symptom) There seems to be enough +1s to justify backport, we just need to be sure it is the right fix ... first step would be to validate on 3.0. Then we can cherry pick and ask for private validation on 2.2/2.1 build.

and

We still need someone to help us track this down: Anyone has an environment where it happens on somewhat regular basis, where we could work with you to collect more logs and experiment? It would be great help. Thanks!

Following up your chat with @LukePulverenti about backporting the fix to 2.2, I have created a reproduction scenario for you: https://github.com/softworkz/ReuseBug

The solution contains a native Linux app and a netcore console app, multi-targeting netcore 2.0, 2.2 and 3.0

This demonstrates:

• works in 2.0
• fails in 2.2
• works again in 3.0

I hope this helps getting the fix backported to 2.2...

[LukePulverenti]

@softworkz thank you !

@karelz Yes it would be great to get this back-ported because ever since the 2.1 release we've had to tell users to shutdown all other upnp or dlna software on the machine in order to prevent this from happening.

[wfurt]

BTW Here is nice reading: https://idea.popcount.org/2014-04-03-bind-before-connect/

[karelz]

@softworkz @LukePulverenti I think we may be dealing with multiple problems here as some people on this thread said that 3.0 does not fix it for them. Either way, we have a repro now, so let's try it -- @tmds or @wfurt will you have time to try it out and reproduce? If we can reproduce in-house, it should be easier for us to track it down. I'd be also interested in the repro result on 2.1.

Thanks @softworkz for repro!!! That is a HUGE step towards root cause and solution. Let's hope we can reproduce it too :)

[softworkz]

@karelz - Yes, ours is about the UDP bug https://github.com/dotnet/corefx/issues/32027 which was correctly fixed for 3.0 by PR https://github.com/dotnet/corefx/pull/32046 and we're hoping to get it backported to 2.2. It's getting a bit embarrassing having to tell users that our software cannot coexist with other DLNA software, especially once they've found out that any other two (non-netcore) applications can do that.. ;-) (even worse is that it has been working in a previous version with netcore 2.0)

Regarding 2.1: It fails with 2.1 as well. I've just updated the repro solution (https://github.com/softworkz/ReuseBug) by adding 2.1 as additional target framework and publishing target.

[tmds]

Thanks @softworkz for repro!!! That is a HUGE step towards root cause and solution. Let's hope we can reproduce it too :)

@karelz @softworkz is talking about a UDP issue https://github.com/dotnet/corefx/issues/32027 which was decided not to be backported: https://github.com/dotnet/corefx/issues/32027#issuecomment-418447086.

The main issue reported here is a TCP issue observed when using HttpClient.

[softworkz]

@karelz @softworkz is talking about a UDP issue dotnet/runtime#27274 which was decided not to be backported: #32027 (comment).

And still we're asking for it. It's a bug - not a "corner case".

The main issue reported here is a TCP issue observed when using HttpClient.

Not quite. We're not the only ones referring to the UDP bug here.

[tmds]

Not quite. We're not the only ones referring to the UDP bug here.

Yes, this is causing confusion, so it's good to make clear the difference. The issue reported here is for HttpClient/TCP, and it was assumed the UDP fix would solve it, which is not the case.

[karelz]

Agreed, this issue is already pretty confusing even without mixing it up with UdpClient. Let's keep this issue specific to HttpClient/TCP (I will update the title). Let's move the discussion about backporting dotnet/runtime#27274 into separate issue please (we can reuse dotnet/runtime#27274 for now) -- BTW: I would be interested in confirmation what exactly are symptoms of UdpClient - in that issue, not here please. Note: So far I believe we have 2 customers hitting it.

[softworkz]

@karelz - Can you move posts? Or should I repeat my information in the other issue?

[karelz]

@softworkz posts cannot be moved, please copy relevant information over. Thank you!

BTW: I hid the UdpClient comments from the thread to avoid further confusion.

[karelz]

Just to reiterate: We still need someone to help us track the HttpClient problem down: Anyone has an environment where it happens on somewhat regular basis, where we could work with you to collect more logs and experiment? It would be great help that would unblock us. Thanks! See instructions from @wfurt above: https://github.com/dotnet/corefx/issues/37044#issuecomment-495425689

[karelz]

@yuezengms @yanrez @arsenhovhannisyan @antoinne85 @blurhkh @EvilBeaver @antonioortizpola @LukePulverenti @sapleu @rrudduck @rbrugnollo @robjwalker @OpenSourceAries I'd like to ask you for 2 favors:

1. Can you please confirm if your repro is truly on HttpClient/TCP and NOT UdpClient? (please confirm you're on HttpClient/TCP by upvoting this reply)
2. Is any one of you in position to collect additional logs and work with us to root cause this problem? We would love to address it, but we have nothing actionable at this moment without help from someone who can hit the problem and can collect additional info. Thanks!

[dmiller02]

We hit this issue on a .net core 2.2 application running on Azure Linux Kubernetes. We tried using .net core 3 and while this improved some of the issues we still consistently hit this error. Our investigations found that the HttpClient wasn't releasing ports despite the clients being disposed. Though when running in windows the client was releasing the ports. We updated our dependency injection to use the IHttpClientFactory and used this to create the HttpClients which fixed our issues.

[stephentoub]

We tried using .net core 3 and while this improved some of the issues we still consistently hit this error. Our investigations found that the HttpClient wasn't releasing ports despite the clients being disposed.

Got a repro you can share?

[dmiller02]

We tried using .net core 3 and while this improved some of the issues we still consistently hit this error. Our investigations found that the HttpClient wasn't releasing ports despite the clients being disposed.

Got a repro you can share?

Sorry, unfortunately not.

These are the pages we found that helped us: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-2.2 https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests

[karelz]

@dmiller02 are you in position to get back into bad state and help us collect some logs?

[dmiller02]

@dmiller02 are you in position to get back into bad state and help us collect some logs?

Shouldn't be too difficult. We have the images for the service in question and can re-create the error. What logging would you need?

[karelz]

@wfurt @stephentoub what kind of logs may help us confirm what is going on, on Linux?

[wfurt]

To begin with, can we collect output of netstat -natu wehn it happens and than run sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535" to see if that improves the situation?

[antonioortizpola]

I have run netstat -natu in our server and it looks like the error is with the WSDL endpoints (the address 172.17.72.150 is a WSDL service).

To work with the WSDL we are adding the service as Transient:

        public static IServiceCollection AddExternalServices(
            this IServiceCollection serviceCollection,
            IConfiguration configuration)
        {
            serviceCollection.AddGrpc();

            serviceCollection.AddTransient<CenamWsAddClaroMicroCredit>();
            return serviceCollection;
        }

And then we use it in the service like this

        public async Task<AddClaroMicroCreditResponse> AddClaroMicroCredit(RequestInfo requestInfo)
        {
            var wsdlServiceClient = new WsdlServiceClient(
                CenamOperationClient.EndpointConfiguration.CenamOperationPort, _cenamOpConfig.CreditsEndpoint);
            var timedWsdlRequestWithLog = new TimedWsdlRequestWithLog(_logger, ServiceName);

            await _cenamWsThrottling.WaitToActionAndIncrement();
            var response = await timedWsdlRequestWithLog.ExecuteLogDurationAndReturnRequest(
                wsdlServiceClient.Endpoint,async () =>
                    await wsdlServiceClient.addClaroMicroCreditAsync(
                        _cenamOpConfig.Entity,
                        requestInfo.Data1,
                        requestInfo.Data2,
                        requestInfo.Data3)
            );
            // work with the result...
        }

The method to log is just a function to log the request and response with the service name

        public async Task<RequestRawAndResult<T>> ExecuteLogDurationAndReturnRequest<T>(ServiceEndpoint serviceEndpoint, Func<Task<T>> action)
        {
            var requestLogAndTime = new RequestLogAndTimeEndpointBehavior();
            serviceEndpoint.EndpointBehaviors.Add(requestLogAndTime);

            T response;
            try
            {
                response = await action();
            }
            catch (Exception e)
            {
                _logger.LogError(e,
                    "Unexpected error requesting to wsdl service '{serviceName}' in {responseTime}ms to '{serviceEndpointAddress}' with body '{request}' responded: {response}",
                    _serviceName,
                    requestLogAndTime.LastResponseTimeInMillis,
                    serviceEndpoint.Address,
                    requestLogAndTime.LastRequestXml,
                    requestLogAndTime.LastResponseXml);
                throw;
            }

            _logger.LogInformation(
                "Request to service '{serviceName}' in {responseTime}ms to '{serviceEndpointAddress}' with body '{request}' responded: {response}",
                _serviceName,
                requestLogAndTime.LastResponseTimeInMillis,
                serviceEndpoint.Address,
                requestLogAndTime.LastRequestXml,
                requestLogAndTime.LastResponseXml);

            return new RequestRawAndResult<T>(response, requestLogAndTime.LastRequestXml, requestLogAndTime.LastResponseXml);
        }

Maybe something inside the wsdl client is causing the issue (like creating the HttpClient by itself), but then, how I could workaround this (this way had no problems at all with core 2.1),

[wfurt]

From the log:

tcp        1      0 172.20.0.2:38736        172.17.72.150:28085     CLOSE_WAIT

That means the server or client did not finish closing the socket. You should also see this with lsof -i -n -p <PID> Can you please do packet capture for few requests @antonioortizpola ? I'm wondering if this happens on every request and it just takes some time to use all port numbers.

I'm not familiar with WSDL code. Can you craft runable repro - just like we got one for the UDP case. I think this also depends on server no closing the socket so we may not be able to reproduce it with only client side.

We do not need to hit the bind error. All we need to reproduce is getting new socket stuck in CLOSE_WAIT state.

[antonioortizpola]

@wfurt sure, I can work in my project by removing all code, just leaving one WSDL client.

I will try to work on this on the weekend (right now I am in the office and have some other tasks), creating a simple solution and make some packet capture with wireshark, then I will let you know what i can find.

[wfurt]

thanks @antonioortizpola . it would be nice to get to bottom of this. Seeing half-closed TCP is certainly clue.

[wfurt]

BTW any chance @antonioortizpola that you can share core dump from time when it is failing? (it does not need to reach port exhaustion - we just need few sockets in half-closed state) It will be large and it can contain any secrets or private data. But if we can work it out I think we would be able to sort this out. If that not possible, we may be able to script dump file processing or I can guide you through sequence to get useful info out.

[antonioortizpola]

Ok, I have the example!!! @wfurt, @karelz, I will make a readme, but the results are clear.

I created two projects, one with core 3 and one with core 2.1, It should be virtualy the same code, but after some stress tests, the version with core 3 does not release the ports:

....
tcp        1      0 172.18.0.3:39787        172.18.0.4:80           CLOSE_WAIT
tcp        1      0 172.18.0.3:34693        172.18.0.4:80           CLOSE_WAIT
tcp        1      0 172.18.0.3:46675        172.18.0.4:80           CLOSE_WAIT
tcp        1      0 172.18.0.3:38431        172.18.0.4:80           CLOSE_WAIT
tcp        1      0 172.18.0.3:45375        172.18.0.4:80           CLOSE_WAIT
tcp        1      0 172.18.0.3:46011        172.18.0.4:80           CLOSE_WAIT
tcp6       0      0 :::80                   :::*                    LISTEN
udp        0      0 127.0.0.11:49014        0.0.0.0:*

While the 2.1 it does

root@1b70aaaed9f2:/app# netstat -natu
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:50051         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:50051         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:50051         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50051           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50051           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50051           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50051           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:50051         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.11:33615        0.0.0.0:*               LISTEN
tcp        0      0 172.18.0.2:35071        172.18.0.4:80           TIME_WAIT
tcp        0      0 172.18.0.2:43885        172.18.0.4:80           TIME_WAIT
tcp        0      0 172.18.0.2:43295        172.18.0.4:80           TIME_WAIT
tcp        0      0 172.18.0.2:33653        172.18.0.4:80           TIME_WAIT
tcp        0      0 172.18.0.2:45709        172.18.0.4:80           TIME_WAIT
tcp        0      0 172.18.0.2:37075        172.18.0.4:80           TIME_WAIT
tcp        0      0 172.18.0.2:36519        172.18.0.4:80           TIME_WAIT
tcp        0      0 172.18.0.2:34009        172.18.0.4:80           TIME_WAIT
tcp        0      0 172.18.0.2:35771        172.18.0.4:80           TIME_WAIT
tcp        0      0 172.18.0.2:46763        172.18.0.4:80           TIME_WAIT
udp        0      0 127.0.0.11:55536        0.0.0.0:*
root@1b70aaaed9f2:/app# netstat -natu
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:50051         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:50051         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:50051         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50051           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50051           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50051           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50051           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:50051         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.11:33615        0.0.0.0:*               LISTEN
udp        0      0 127.0.0.11:55536        0.0.0.0:*
root@1b70aaaed9f2:/app#

It is a little late now, but tomorrow I can upload the project and give you access so you can run it yourselves

[antonioortizpola]

Ok, I have the repo, I invited @karelz and @wfurt, I hope this can help with something, please let me know if I can help with something else.

[wfurt]

thanks @antonioortizpola, I will take a look. Are you suggesting 3.0 fixes the problem?

[antonioortizpola]

@wfurt nooo, the problem did not happen in core 2.1, but it is happening in 2.2 and 3, and you are welcome, again, if I can help in something else just let me know.

You can simply try to update the app in 2.1 to 2.2 and the problem will appear.