Reduce likelihood of server reaching capacity on inbound connections

[gkc]

Lead: @gkc

Describe the bug

• There is currently some reaping of inbound connections in the at_secondary_server, as follows:
  • When the listener asks InboundConnectionManager to create an inbound connection, the manager checks its 'hasCapacity()' method
  • hasCapacity() will attempt to close all 'inValid' inbound connections - i.e. connections where _isIdle() || getMetaData().isClosed || getMetaData().isStale;
  • connections are deemed idle once the configurable duration inboundIdleTimeMillis has passed without activity. This currently defaults to 600000 i.e. 600 seconds i.e. 10 minutes
• However we have no real defense against using-up-all-available-inbound-connections DoS attacks on individual secondaries whether deliberate or as an inadvertent result of badly-written clients
• See #502 for some additional context

Expected behavior Needs further analysis. Some initial thoughts:

• As with #502 for outbound clients, the current hard limit on number of inbound clients should probably become a low-water-mark.
• Until low-water-mark is reached, we can continue with the current approach
  • We can also 'prefer' to reap connections in this order: unauthenticated, authenticated but from some other atSign, authenticated and from this secondary's atSign
• While above low-water-mark:
  • If no inbound clients can be reaped because they are all active, then we can add additional inbound clients up to a high-water-mark
  • as we approach the high-water-mark, we can start to reduce inboundIdleTimeMillis from its configured value down to an absolute minimum of, say, 1 second
  • and also logging WARNING or SEVERE log messages as we approach the high-water-mark, so that some appropriate action can be taken whether by human or automation (e.g. increase the limits for this secondary, block the originating IP addresses, etc)

As we implement what is required for this and for #502, we will need to also invest in client-side handling of outbound connections. There is some discussion of what this looks like in #502 - basically it means if we are keeping outbound connections open from the clients, then we need to have graceful retries for when the server end has had to kill them

Additional context As the network of connections between atSigns grows, we need to have policies which will optimise for the "normal" number of inter-server connections but will gracefully handle much larger numbers

[gkc]

@cconstab @cpswan @VJag @murali-shris @kalluriramkumar @nickelskevin Interested in any initial comments or thoughts you have on this

[gkc]

Marked this as P1 since we are seeing issues related to handling of connections in the secondary servers.

[VJag]

• Right now server does not allow connections from a source if it is in the blocked list. However, it does not help us with preventing a DoS attack

• code for this is in InboundConnectionManager

• One way is to have a rate limit for the number of connections a single @sign can take

• Or start with a *(everyone) for the blocked list and allow as we go with the rate limit

[gkc]

This still requires me to complete writing the end to end tests. 2SP for the remaining work.

[gkc]

Remaining work on this issue: Merge #583

Filed new issue #587 for additional work

[gkc]

Done, after a couple of follow-on PRs