Issue with .net core connecting to Remote Microsoft Sqlserver

[itdeepdive]

Environment:

Dot.net Core Version: .net core 2.0.6 Operating System: CentOS Linux release 7.4.1708 (Core) Sqlserver Hosted on Windows: Microsoft SQL Server 2017 (RTM) - 14.0.1000.169 (X64)

Same Error Exists Using Docker microsoft/dotnet 2.0

Error:

Application startup exception: System.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught) ---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer

   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStreamInternal.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStreamInternal.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, LazyAsyncResult asyncResult)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(Boolean buffer)
   at Microsoft.EntityFrameworkCore.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.<_TrackEntities>d__17`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Example.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env) in /tmp/em3_Example/Example/Startup.cs:line 59
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.Internal.AutoRequestServicesStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
ClientConnectionId:020ebc45-ac5e-4119-a789-865174be336b
crit: Microsoft.AspNetCore.Hosting.Internal.WebHost[6]
      Application startup exception
System.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught) ---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStreamInternal.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStreamInternal.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, LazyAsyncResult asyncResult)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(Boolean buffer)
   at Microsoft.EntityFrameworkCore.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.<_TrackEntities>d__17`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Example.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env) in /tmp/em3_Example/Example/Startup.cs:line 59
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.Internal.AutoRequestServicesStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
ClientConnectionId:020ebc45-ac5e-4119-a789-865174be336b

Unhandled Exception: System.Data.SqlClient.SqlException: A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught) ---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStreamInternal.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStreamInternal.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, LazyAsyncResult asyncResult)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
   --- End of inner exception stack trace ---
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(Boolean buffer)
   at Microsoft.EntityFrameworkCore.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.<_TrackEntities>d__17`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Example.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env) in /tmp/em3_Example/Example/Startup.cs:line 59
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.Internal.AutoRequestServicesStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Example.Program.Main(String[] args) in /tmp/em3_Example/Example/Program.cs:line 18
Aborted

[EDIT] Fix exception formatting by @karelz

[karelz]

How often does it happen? Is it 100% repro or intermittent? Does it happen right away, or after some time? Did you do additional network tracing on client and/or server to confirm it is not server closing the connection?

[karelz]

Could you try .NET Core 2.1 RC? (released earlier this week as "go-live")

[akasianenko]

Have just tried .NET Core 2.1 RC (dotnet:2.1-aspnetcore-runtime) and got the same error. Reproduces permanently. The very same connection string from non-core dotnet windows application works fine. SQL Server in question is 2016 SP1, and for SQL Server 2014 SP2 connection works just fine. I would really appreciate any ideas, thanks.

[itdeepdive]

@karelz This Happens all the time. I tested it through 2.1.0-rc1-final.

I get the exact same error.

Thanks Trilok

[MaksymProkopiv]

We tested on .NET Core 2.0.6 and 2.1 RC with the same results: [Exception] SQL Server 2008 (SP1) - 10.0.2710.0 (X64)
[OK] SQL Server 2014 - 12.0.2000.8 (X64) [OK] SQL Server 2014 (SP2) (KB3171021) - 12.0.5000.0 (X64) [Exception] SQL Server 2014 (SP2-CU8) (KB4037356) - 12.0.5557.0 (X64) [Exception] SQL Server 2014 (SP2-CU10-GDR) (KB4052725) - 12.0.5571.0 (X64) [Exception] SQL Server 2016 (SP1) (KB3182545) - 13.0.4001.0 (X64)

[karelz]

Do you have a repro you can share with us? Did you try to look at network traces? (Fiddler, Wireshark)

@keeratsingh @David-Engel @AfsanehR any ideas?

[AfsanehR-zz]

Hi @itdeepdive , will start investigating this and will update you with more updates here. Thanks!

[logankp]

I also recently started experiencing this issue. It only seems to happen in Linux containers for me. Works just fine using .Net Core on Windows.

[EltonCarreiro]

The same happens here as @logankp described.

[AfsanehR-zz]

@EltonCarreiro and @logankp thank you for confirming this issue. I just successfully ran a sample application to read data from a remote SQL Server on my Windows machine from an application on a CentOS machine.

# dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.300
 Commit:    adab45bf0c

Runtime Environment:
 OS Name:     centos
 OS Version:  7
 OS Platform: Linux
 RID:         centos.7-x64
 Base Path:   /root/dotnet/sdk/2.1.300/

Host (useful for support):
  Version: 2.1.0
  Commit:  caa7b7e2ba

.NET Core SDKs installed:
  2.1.300 [/root/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0 [/root/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0 [/root/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.0 [/root/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Although I had to download the dotnet-sdk-2.1, since using package manager I am getting issues as mentioned on this Github Issue

As @karelz mentioned, do you see this happening right away of after some time? A repro would be helpful to better investigate the issue. Thanks.

[EltonCarreiro]

@AfsanehR thank you for the quick response.

The application is deployed as SCD (using 2.0.6), so I don't have the SDK installed on my Linux server (I don't know if it can change something).

Another difference is that my app is hosted on an Linux Ubuntu 16.04.2 LTS

dotnet --info

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.5
  Build    : 17373eb129b3b05aa18ece963f8795d65ef8ea54

The exception is thrown according to the timeout specified in the connection string, so if I specify 30 seconds, that's the time it takes.

My problem occur exactly when I try to connect to the SQL Server, so the code doesn't have anything different.

public IDbConnection CreateNewConnection()
{
    return new SqlConnection(Configuration["ConnectionStrings:SQLServerDP"]); // throws
}

And as you might expect, at my appsettings.json I have the connectionstring

"ConnectionStrings": {
        "SQLServerDP": "Data Source={SERVER HERE}\\{INSTANCE HERE};Initial Catalog={CATALOG NAME HERE};User ID={USER HERE};Password={PASSWD HERE};Max Pool Size=1000;Connect Timeout=300;"
}

So, as we can see, according to the above connectionstring, it will take 300 seconds to throw the exception.

If you still want a reproduction please let me know and I'll send one ASAP.

[logankp]

This issue is a bit challenging for me to reproduce: it occurs when connecting to SQL Server 2014 (12.0.5207.0) on one server only (so far) but is consistent. I'm able to connect to a 2017 instance on the same server. I've installed SQL Server 2014 on another machine and it can connect to that just fine.

I'm using the microsoft/aspnetcore container and I've tried both EF Core 2.0 and 2.1. The issue occurs on both versions of EF.

[JanR1]

Hello, Having the same problem.

System.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught) ---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer 6/13/2018 5:46:03 PM at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size) 6/13/2018 5:46:03 PM --- End of inner exception stack trace ---

However - It seems to not be tied with exact version of SQL Server. We have 2016 SP1 CU7 (13.0.4466.4) in PROD(Phys Cluster), TEST(VM) and multiple DEV machines (VM). Every server works except the TEST (VM). However, From the exact same container I was able to successfully connect with SQLCMD.

[AfsanehR-zz]

@JanR1 @logankp @EltonCarreiro Thank you all for your responses. At this point, we are unable to repro the issue. Is it possible for you to share the network traces with us?

[EltonCarreiro]

Sorry for the late response.

After the @JanR1 report, sounds like a network related issue.

Since we (in the company) were getting problems with SQL Server and Linux, we are temporarily running the app on a windows machine, so this issue was put aside until the solution is found.

I'll get back to this question ASAP and record the network trace to put it in here.

[logankp]

If someone can point me to some documentation on providing a network trace I can try to provide one as well.

[EltonCarreiro]

I would also appreciate some doc/instructions as @logankp requested, I used to enable tracing just at .NET Framework apps, don't know how to do that using ASP.NET Core.

[rzachariah]

I'm seeing an error that looks similar.

Exception="System.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 35 - An internal exception was caught) ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer    --- End of inner exception stack trace ---    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) --- End of stack trace from previous location where exception was thrown ---    at System.Data.SqlClient.SNI.SslOverTdsStream.ReadInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)    at System.Net.Security.SslStreamInternal.<FillBufferAsync>g__InternalFillBufferAsync|38_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)    at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)

Environment: Docker on Linux Runtime: Both of FROM microsoft/dotnet:2.1-aspnetcore-runtime FROM microsoft/dotnet:2.1-runtime

[David-Engel]

@logankp @EltonCarreiro To capture a network trace, you can use netsh: https://blogs.technet.microsoft.com/yongrhee/2012/12/01/network-tracing-packet-sniffing-built-in-to-windows-server-2008-r2-and-windows-server-2012/ or https://blogs.msdn.microsoft.com/canberrapfe/2012/03/30/capture-a-network-trace-without-installing-anything-capture-a-network-trace-of-a-reboot/

Or a tool like Wireshark: https://support.symantec.com/en_US/article.TECH110371.html

If you are on Linux, there are similar tools available.

[MaksymProkopiv]

I collected tcpdump. @AfsanehR could you please provide any private channel how I can share it with you?

[AfsanehR-zz]

@MaksymProkopiv Please find my email address at my profile.

[EltonCarreiro]

@AfsanehR have you got the data collected by @MaksymProkopiv ? Still need it?

[MaksymProkopiv]

Just find the reason in our case. SQL-client wasn't able to build certificate chain for SQL Server certificate, as it was issued by our Enterprise PKI. Authority Information Access path of Server certificate was pointing to internal server, that wasn't accessible from docker container.

[AfsanehR-zz]

@EltonCarreiro we did collect the data from @MaksymProkopiv , however, it would be still useful for us to have your network trace as there could be other reasons in your case.

@MaksymProkopiv thanks for reporting back to the issue.

[AfsanehR-zz]

@EltonCarreiro @logankp @rzachariah Please share your network traces if your issue still does exist.

[dixonte]

I am also having this issue. However I feel like I may be in an odd edge case, since I'm running dotnetcore-sdk v2.1.105 on a Gentoo machine and trying to connect to SQL Server 2005 (yes, I know, I know, it can't be helped for now).

[logankp]

I'll try to get a network trace later today or next week. My company has deprioritized fixing this and I haven't had much time.

[logankp]

@AfsanehR Emailed you the network trace

[ilzenzo]

Hello, any news on this issue?

[danmoseley]

@afsanehr

[AfsanehR-zz]

@logankp sorry for delay in response. Are you seeing the same error message as mentioned at the beginning of this issue? From what I can see from the traces you provided, no connection is established at all, since the client is only sending SYN or ACK or PSH,ACK packets without receiving any SYN,ACK from server, therefore, no TCP handshake happens at all. This could be due to a firewall blocking those packets.

@ilzenzo Are you receiving the same error? This is not reproable on our end and it would help if you could also provide network trace.

[logankp]

I was getting the same error message as above. It's interesting that it wasn't connecting at all. I haven't re-tested since I got the network trace so I don't know if it's still an issue.

[WeihanLi]

I met the same error, it seemed that there's too many connections, limit your concurrent size may help

[codesphinxx]

I am currently experiencing this issue, has anyone found a solution to this?

[Sevsoad]

@AfsanehR Having same issue on 2.1.4 sdk, trying to connect from redhat to sql server 2005. Same setup works fine with sql server 2011.

Connection fails after sending pre-login tds packages.

[AfsanehR-zz]

@codesphinxx what version of SQL Server are you using? @Sevsoad Sql Server 2005 is out of support. Please use an updated version. Link

[YagoAzevedo]

Some news?

[David-Engel]

@YagoAzevedo Reading through the various comments, it seems all the problems where someone was able to identify an underlying cause are related to network configuration or certificate chain configuration issues. Nothing yet pointing to an underlying issue in SqlClient itself.

[davaug2]

I have the same problem. Scenario:

Dotnet core version: 2.2 Operarion system: Ubuntu 16.04 SqlServer 2017 on linux Ubuntu

Everything works fine until you have a lot of concurrent requests, the error starts with about 50 concurrent requests. From that point 20% of the requests return error, timeout when trying to connect, another 30% the network transport error.

I noticed that the problem occurs when trying to open connection with database, using Sqlconnection.

A few minutes later 100% of the requests return an error. So I need to restart the application.

I hosted the same application as Windows service in Win Server 2019, the problem was miraculously resolved.

I would like to continue using linux, but it is not possible for now.

[ashtonian]

using .netcore 2.2 docker on linux service plan connecting to azure sql server instance and we are getting lots of reset by peer errors.

[sirdarkat]

Same as pretty much everyone else. Linux Docker Container Running 2.2 getting reset by peer errors. Do we have any updates on a resolution for this issue or is this just a feature of .Net Core?

[David-Engel]

@jetdev @Ashtonian @sirdarkat Connection reset by peer errors are very generic and can have many different underlying causes. The original description in this issue talks about an immediate error on startup. @jetdev talks about errors when concurrent requests get high. The underlying problem is almost certainly different in these two situations.

@jetdev Your scenario sounds like you many actually be hitting a bug in SqlClient on Linux since your issue does not occur on Windows and it only occurs under load. I would encourage you to file your own issue to get more specific attention.

@Ashtonian and @sirdarkat I wonder if your scenarios may be helped by #33024 which adds TCP KeepAlives to non-Windows platforms in .NET Core 3.0. Azure is pretty aggressive about killing inactive network connections. Give the latest 3.0 preview a try. Other than that, you'll need to provide more details about your specific situations (preferably by filing your own issues).

[sirdarkat]

@David-Engel Thank you for the response. I will give the preview version a try and see if the keep alive helps out if not I will do as you suggest and open an issue with a bit more back story on what I'm doing and what I'm seeing. Once again thank you.

[namdinh0312]

I have same issue as @jetdev said. Dotnet core : 2.2 Operarion system: Ubuntu 16.04 MongoDB I use docker compose to scale many web services. All Web services connect to one VPS MongoDB.

System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) at System.Threading.Tasks.ValueTask1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) --- End of stack trace from previous location where exception was thrown --- at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytesAsync(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync() at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync() at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(Int32 responseTo, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken) at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken) at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol1 protocol, CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.CommandOperationBase1.ExecuteProtocolAsync(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.ReadCommandOperation1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.FindCommandOperation1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.FindOperation1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken) at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation1 operation, CancellationToken cancellationToken) at MongoDB.Driver.MongoCollectionImpl1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation1 operation, ReadPreference readPreference, CancellationToken cancellationToken) at MongoDB.Driver.MongoCollectionImpl1.UsingImplicitSessionAsync[TResult](Func2 funcAsync, CancellationToken cancellationToken) at MongoDB.Driver.IAsyncCursorSourceExtensions.FirstOrDefaultAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken) at Ryshin.Reponsitories.MongoReponsitories.TagReponsitory.Get(String id) in

[David-Engel]

I have the same problem. Scenario:

Dotnet core version: 2.2 Operarion system: Ubuntu 16.04 SqlServer 2017 on linux Ubuntu

Everything works fine until you have a lot of concurrent requests, the error starts with about 50 concurrent requests. From that point 20% of the requests return error, timeout when trying to connect, another 30% the network transport error.

I noticed that the problem occurs when trying to open connection with database, using Sqlconnection.

A few minutes later 100% of the requests return an error. So I need to restart the application.

I hosted the same application as Windows service in Win Server 2019, the problem was miraculously resolved.

I would like to continue using linux, but it is not possible for now.

@Wraith2 This is the issue I mentioned that made me think of you. To me, this indicates that under load, something in the managed SNI stack is not up to snuff compared to the native SNI on Windows. If you are interested, you could try this on Windows using the managed SNI trick and see if it repros, or learn some Linux. 😄

[Wraith2]

I've been doing quite a lot of work to bring the managed SNI implementation performance and stability closer to the native version used on windows. There are a couple of blocking issues we've talked about which prevent some work but once those are cleared I'll continue. If I can repro this I'll certainly try to fix it.

[sirdarkat]

@David-Engel Just a little update (I know this issue isn't probably 100% the same as mine but not sure I need to open a new one just yet). The preview version didn't fix the issue so I became a little what obsessed with this. Bashed into the docker container and ran netstat -nap and found a ton of Closed_Wait Sockets were pilling up. So I reworked all of the HttpClient items to singletons flipped over to HttpClientFactory to create them and rewroked the EF Core code Im using. Finally got all of my sockets down to only 4 ever being open at a time and they are the 4 currently being used. Didn't fix the issue either but I will say did make me happy to get down to only 4 sockets being open. I'm still not sure what is causing the Connection Reset to occur. SImiliar code on a window server doesn't seem to create these errors only seem to occur when running in linux. They also only seem to occur when a heavy amount of requests are going through the linux code. (Similiar to the comment you have quoted above but I don't have to reset it eventually stops failing and starts working again just fine). I'm not to the point of the quoted comment. The amount of dropped requests are annoying but I have enough retry mechs elsewhere that they are just annoying me not forcing me to give up on .Net Core and Docker just yet.

[divega]

As recently announced in the .NET Blog, focus on new SqlClient features an improvements is moving to the new Microsoft.Data.SqlClient package. For this reason, we are moving this issue to the new repo at https://github.com/dotnet/SqlClient. We will still use https://github.com/dotnet/corefx to track issues on other providers like System.Data.Odbc and System.Data.OleDB, and general ADO.NET and .NET data access issues.

[jagillispie]

We saw some similar issues and used this thread as we were troubleshooting. We have Ubuntu running in Cloud Foundry. We had all certs loaded that were used by the destination SQL server but still got the timeouts with some successes. What was found is a firewall was blocking access to the CRL lists for the internal CA's used by the remote SQL server. Once we allowed the CRL traffic from our Ubuntu instances the timeouts stopped occurring.

[aarro]

We saw some similar issues and used this thread as we were troubleshooting. We have Ubuntu running in Cloud Foundry. We had all certs loaded that were used by the destination SQL server but still got the timeouts with some successes. What was found is a firewall was blocking access to the CRL lists for the internal CA's used by the remote SQL server. Once we allowed the CRL traffic from our Ubuntu instances the timeouts stopped occurring.

Some more info on this...

The error around this changes from dotnet core 2.0 and 2.1. The 2.0 error is a little more helpful, honestly.

2.0 A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught) ---> System.IO.IOException: The write operation failed, see inner exception. ---> System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Connection reset by peer.) ---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer

2.1 System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Connection reset by peer.) ---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer

TrustServerCertificate=true; had no effect on this. The cert was not configured in the MSSQL instance but on the server itself and was involved at a low level. A packet trace showed that the login succeeded with SQL but then the app would "hang", apparently because the first CA server in the cert was unreachable and dotnet behind the scenes was spinning, waiting on response? As @jagillispie mentioned, opening port 80 to the CA server resolved the issue.

If there is logging around this, where we could have seen the failure to connect to the CA server, I cannot find it. It would be super helpful to be able to see the connection failure to the CA as a useable exception or logged item, whether that be in the exception bubbled up from SqlClient or even trace/debug level logging to console.