search

cnblogs / EnyimMemcachedCore

.NET Memcached client. Available on https://www.nuget.org/packages/EnyimMemcachedCore
Apache License 2.0
162 stars 45 forks source link

Sockets being forcibly closed after failure to get socket from pool. #167

Closed Kilowhisky closed 2 years ago

Kilowhisky commented 2 years ago

Hi,

I'm using this library to get/set keys in a background worker thread on a .net core application that is consuming a kafka feed. I seem to be getting errors with sockets being forcibly closed after a failure to acquire a connection from the pool happens. Below is an output from the log. What is interesting is in between the chunks of socket failures there are thousands of messages that get successfully processed.

Ive tried switching from binary to text protocol but that didn't appear to help.

For reference i'm using AWS ElastiCache Memcached instance which is running 1.6.12 with encryption in transit set to ON.

Any ideas what could be going on here? 

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Pool has been inited for 10.134.72.121:11211 with 5 sockets
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedInitPool-cost: 1292.0375ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool on node '10.134.72.121:11211'
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket d4e26e4d-49db-49bd-961a-7cf785e6243e was reset
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket was reset. d4e26e4d-49db-49bd-961a-7cf785e6243e
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: Failed to ExecuteOperation on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetHelper.ReadItem(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket d4e26e4d-49db-49bd-961a-7cf785e6243e
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket cf89a5db-87e4-4af4-98c4-99abe15e0029 was reset
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket was reset. InstanceId cf89a5db-87e4-4af4-98c4-99abe15e0029
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: pooledSocket.WriteAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Enyim.Caching.Memcached.Protocol.Text.StoreOperation.ReadResponseAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: IOException occurs when ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation) on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.StoreOperationBase.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperationAsync(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket cf89a5db-87e4-4af4-98c4-99abe15e0029
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool on node '10.134.72.121:11211'
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket 955aaf74-cf3e-4ee7-a0dc-47a0c8694169 was reset
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket was reset. 955aaf74-cf3e-4ee7-a0dc-47a0c8694169
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: Failed to ExecuteOperation on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetHelper.ReadItem(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket 955aaf74-cf3e-4ee7-a0dc-47a0c8694169
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool on node '10.134.72.121:11211'
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket faa3b644-f370-4bfc-bfcf-bc698782eca2 was reset
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket was reset. faa3b644-f370-4bfc-bfcf-bc698782eca2
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: Failed to ExecuteOperation on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetHelper.ReadItem(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket faa3b644-f370-4bfc-bfcf-bc698782eca2
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket 4d5d97a9-c0d4-4f0e-b204-3e681219ce44 was reset
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Socket was reset. InstanceId 4d5d97a9-c0d4-4f0e-b204-3e681219ce44
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: pooledSocket.WriteAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Enyim.Caching.Memcached.Protocol.Text.StoreOperation.ReadResponseAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: IOException occurs when ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation) on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.StoreOperationBase.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperationAsync(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket 4d5d97a9-c0d4-4f0e-b204-3e681219ce44
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Warning: Marking node 10.134.72.121:11211 as dead
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Node 10.134.72.121:11211 is dead.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Starting the recovery timer.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Timer started.

The thread 0xe920 has exited with code 0 (0x0).
The thread 0x4dc4 has exited with code 0 (0x0).
The thread 0xcd94 has exited with code 0 (0x0).
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Checking the dead servers.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Dead: 10.134.72.121:11211

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Ping ok.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Reinitializing the locator.

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: deadCount == 0, stopping the timer.

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool on node '10.134.72.121:11211'
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 51.6832ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: Failed to ExecuteOperation on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetHelper.ReadItem(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket 1f5533e5-fb5a-4030-b44e-25c3d5384cea
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211

Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 52.6924ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: pooledSocket.WriteAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Enyim.Caching.Memcached.Protocol.Text.StoreOperation.ReadResponseAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: IOException occurs when ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation) on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.StoreOperationBase.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperationAsync(IOperation op)

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket df0d6736-e55b-4f58-bbd2-097e365ceeb8
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool on node '10.134.72.121:11211'
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 43.0492ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: Failed to ExecuteOperation on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetHelper.ReadItem(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket f7743474-5bcf-44fa-b484-9bb1c6ccfdaf
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool on node '10.134.72.121:11211'
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 47.9131ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: Failed to ExecuteOperation on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetHelper.ReadItem(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket 033ee22a-b0c9-4599-8c9b-b2fc2c026942
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 47.222ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: pooledSocket.WriteAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Enyim.Caching.Memcached.Protocol.Text.StoreOperation.ReadResponseAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: IOException occurs when ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation) on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.StoreOperationBase.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperationAsync(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket 363221b8-1db9-47aa-9b4c-42f7a9834f4a
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Warning: Marking node 10.134.72.121:11211 as dead
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Node 10.134.72.121:11211 is dead.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Starting the recovery timer.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Timer started.

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Checking the dead servers.

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Dead: 10.134.72.121:11211

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Ping ok.

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Reinitializing the locator.

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: deadCount == 0, stopping the timer.

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool on node '10.134.72.121:11211'
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 47.4067ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: Failed to ExecuteOperation on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetHelper.ReadItem(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket f13dcb42-3e16-4983-bdb0-e6d284535f7f
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 46.9123ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: pooledSocket.WriteAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Enyim.Caching.Memcached.Protocol.Text.StoreOperation.ReadResponseAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: IOException occurs when ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation) on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.StoreOperationBase.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperationAsync(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket a164294a-c218-42ec-a568-59dac935a0e1
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool on node '10.134.72.121:11211'
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 49.6894ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: Failed to ExecuteOperation on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetHelper.ReadItem(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket 4304b486-378c-4350-a3fb-fa64ab12d56b
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False

Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool on node '10.134.72.121:11211'
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 46.3733ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: Failed to ExecuteOperation on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetHelper.ReadItem(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.GetOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket fc7e3bae-2e1c-435b-af38-55f426b17441
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): False
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Acquiring stream from pool. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Could not get a socket from the pool, Creating a new item. 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Information: MemcachedAcquire-CreateSocket: 51.0657ms
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Done.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: pooledSocket.WriteAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Enyim.Caching.Memcached.Protocol.Text.StoreOperation.ReadResponseAsync...
Enyim.Caching.Configuration.MemcachedClientConfiguration: Error: IOException occurs when ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Text.StoreOperation) on 10.134.72.121:11211

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at Enyim.Caching.Memcached.PooledSocket.ReadByte()
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadLine(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.TextSocketHelper.ReadResponse(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Text.StoreOperationBase.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponseAsync(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperationAsync(IOperation op)
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Releasing socket 0c0a0c6c-08f8-48ad-a077-a629deac5414
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Are we alive? True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Mark as dead was requested for 10.134.72.121:11211
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: FailurePolicy.ShouldFail(): True
Enyim.Caching.Configuration.MemcachedClientConfiguration: Warning: Marking node 10.134.72.121:11211 as dead
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Node 10.134.72.121:11211 is dead.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Starting the recovery timer.
Enyim.Caching.Configuration.MemcachedClientConfiguration: Debug: Timer started.
Kilowhisky commented 2 years ago

Turns out Encryption in Transit was causing it to go kaboom. I recreated the ElastiCache node with encryption off and its working fine now.