Connection reset by peer

[stefanprodan]

I'm having the following situation: at app start I'm opening a connection and use it as singleton in my code. Before I run a query, I check if the connection is still open , if not, I call reconnect.

After 20 minutes of inactivity, the conn.Open returns true but when I run a query I'm getting this error:

Write Query failed for Token 6. Exception: Unable to write data to the transport connection: Broken pipe.
Response Pump: IOException - Unable to read data from the transport connection: Connection reset by peer. The connection can no longer be used. ResponsePump is preparing to shutdown.
Cleaning up Response Pump awaiters for rdb-proxy:28015

How can I know when to call reconnect if conn.Open returns true even if the pipe is broken?

Thanks

[bchavez]

Hi @stefanprodan

What OS and runtime?

[stefanprodan]

The RethinkDb cluster in running on Docker Swarm, RethinkDb version is 2.3.4. The aspnetcore app is also running in the same docker swarm.

[stefanprodan]

Here is the connection logic and I'm using it like this

[bchavez]

Hey @stefanprodan ,

Looking at the code path, it appears we have: Connection.Open -> SocketWrapper.Open -> Socket.Connected

• Connection.Open is our code.
• SocketWrapper.Open is our code.
• Socket.Connected is the provided by the .NET runtime/framework.

It's straight through call down to Socket.Connected without anything fancy.

Checking the source code for Socket here:

// Gets the connection state of the Socket. This property will return the latest
// known state of the Socket. When it returns false, the Socket was either never connected
// or it is not connected anymore. When it returns true, though, there's no guarantee that the Socket
// is still connected, but only that it was connected at the time of the last IO operation.
public bool Connected

Not much we can do about it unless there's some other Socket API call to determine connectedness that we should be using...

I'm open to any ideas ..... :thought_balloon:

:thoughtballoon: :calendar: [**"Something's missing from the mind... Think about it everyday since '95."**_](https://www.youtube.com/watch?v=S3vDBPO0FcQ)

[stefanprodan]

Maybe you could change the CheckOpen function to ping the RethinkDb server instead of just checking the Socket state. Or better create a new method named Ping that will do the round-trip to server and back.

[bchavez]

The only thing I can think of is, try{}catch{} the query and reconnect if there's some kind of exception.

ie:

try{
 //run query
}
catch{
  //_now_ that you've attempted an IO operation
  //could you check here? maybe will conn.Open be false this time?
  //if so, then attempt to reconnect
}

As for ping, I think we have something currently Connection.Server() which does ping the server about server meta info like server name and stuff. Could you wrap that one in an extension method?

I'm still not sure how I feel about a dedicated Ping method, though. It seems we'd still be in the same situation (but less so) because wouldn't know if the connection is legitimately open or not until we actually try to write something (ie: the actual query).

Let me know your thoughts....

:thoughtballoon: :mag: [**"I want to know... what you're thinking..."**_](https://www.youtube.com/watch?v=ijAYN9zVnwg)

[stefanprodan]

OK but If I do that then I will lose the changefeed items.

[bchavez]

Hey @stefanprodan , looking at the connection logic, my gut feeling is that it's the wrong singleton-pattern for reconnecting. Try the try{}catch{ open? reconnect} pattern.

In what way do you lose changefeed items with try{}catch{}? Could you explain more? Do you have some more code I can look at?

[stefanprodan]

Here is my changfeed code

[stefanprodan]

I see that this problem is related to TCP keepalive but looking at the ConnectInternalAsync you are using Keep Alive ...

[bchavez]

I'm not sure I see how it breaks the ability to _use_ changefeed items. Are you thinking that when a connection breaks, you might lose changefeed items that were due inbound? I don't think there's any way around that.

IIRC, RethinkDB does not support resumable change feeds yet: https://github.com/rethinkdb/rethinkdb/issues/3471

:cloud: :sunny: Bassnectar - Chasing Heaven - Into the Sun

[stefanprodan]

Ah OK, I will stop using change feeds then.

I will switch to try{}catch{ open? reconnect} pattern, thanks a lot for your help.

[bchavez]

If you're looking for a reliable feed that can survive TCP disconnection, I think you have to do a little more application level work. Maybe this will help:

https://github.com/neumino/rethinkdbdash/issues/121#issuecomment-237128712

:cat: :bee: _"Orange buttercat... Chasing after... The crazy bee..."_

[stefanprodan]

I will try that. Thanks again