Drop or reject lightweight connections

[avieth]

This is similar to haskell-distributed/distributed-process#434 but instead of dropping a bundle of connections to a peer, I'd like to be able to drop or reject one lightweight connection. This could be useful in case, for instance, a node is under heavy load and would like to continue to accept data on existing connections (perhaps some more data is needed) but reject new connections until the load is lessened.

Seems to me this would require a handshake when setting up lightweight connections. Currently if there's already a heavyweight connection to a peer then one simply sends a control code and then the connection is judged to be up and running, no response is expected. To allow rejection of the connection, the connector would have to wait for a yes or no response from the peer. Maybe this extra latency is unacceptable?

[qnikst]

I really like ability to reject connections. But it seems that ability to break connections is a bit more flexible without introducing any new protocol on n-t-tcp layer. If you have ability to break connections you can:

a. Break all new connections that are unwanted. This introduce extra costs that some handshake procedure is run anyway even if you do not want it.

b. Ability to introduce any handshake/protocol you want in your specific project. This can be used for checking if this is connection is wanted, or redirection to other host if it's under a heavy load or for authentication, or check of client version. In this case if something goes bad you may decide to drop connection. Good thing here is that you can have this functionality generic for any transport, for example check algorithm correctness using n-t-inmemory.

As for latency, I'm not very concerned about additional latency especially if you pay that once per heavyweight connection creation. Also I'm not concerned much at price that comes with additional functionality. Though with this approach I'm not sure I see how it will fits nicely in general case.

P.S. I'm not sure if we should enforce this politics, but we found it's very useful to have end-to-end protocol, where internal protocols are as simple as possible and at a cost of more complex endpoints, that have to make more decisions, tolerate problem, etc. This approach may lead to more complex code on endpoints, but appear to be more general and flexible. And approach in haskell-distributed/distributed-process#434 is closer to end-to-end approach. Also this may not work great if your clients are is not from the trusted codebase.

[edsko]

Making establishing a light-weight connection a synchronous operation (having to wait for some kind of confirmation message) could have quite a significant performance impact on Cloud Haskell code, which establishes potentially very large numbers of lightweight connections.

Given that message delivery is not guaranteed anyway, we could even just use QDist abstraction from haskell-distributed/network-transport-tcp#46 to implement this, simply by silently dropping connections that were rejected.

[dcoutts]

I think having every lightweight connection require a handshake would be not a good idea. It adds overhead everywhere, and would make it unsuitable for Cloud Haskell. That said, the ability to push back selectively is clearly useful. So perhaps the ability to explicitly notify the other side that we're breaking/dumping selected inbound connections is useful. Otherwise we need per-connection flow control which gets difficult. Arguably applications that need per-connection control flow should do it on top of each lightweight connection.

[qnikst]

I thought that was about per-heavyweight connection decision..