Incoming connections optimizations (MHD tuning and more)

[kzangeli]

FGM: originally this issue was named "Control number of simultaneous incoming connections" but I think we should extend scope to other enhancements related with the way incoming connections are managed (mainly in MHD).

Right now the broker accepts ALL incoming connections, creating a thread for each connection. We have seen that it is pretty easy to choke the broker, sending in a huge amount of requests.

Hopefully microhttpd has already implemented some kind of protection and we just need to use it. If not, we'll have to implement something ourselves.

If MHD helps us out: Effort: 1 man day

Else ... Effort: 3 man day

[kzangeli]

MHD seems to help us with this:

https://www.gnu.org/software/libmicrohttpd/manual/libmicrohttpd.html

clients can specify resource limits on the overall number of connections,
number of connections per IP address and memory used per connection
to avoid resource exhaustion.

[fgalan]

Also related with this issue, we have found that the footprint used by MHD in incoming connections could be dangerously high. In particular, we are using now this initialization:

      mhdDaemon = MHD_start_daemon(MHD_USE_THREAD_PER_CONNECTION | MHD_USE_SSL, // MHD_USE_SELECT_INTERNALLY
                                   htons(port),
                                   NULL,
                                   NULL,
                                   connectionTreat,                     NULL,
                                   MHD_OPTION_HTTPS_MEM_KEY,            httpsKey,
                                   MHD_OPTION_HTTPS_MEM_CERT,           httpsCertificate,
                                   MHD_OPTION_CONNECTION_MEMORY_LIMIT,  memoryLimit,
                                   MHD_OPTION_SOCK_ADDR,                (struct sockaddr*) &sad,
                                   MHD_OPTION_NOTIFY_COMPLETED,         requestCompleted, NULL,
                                   MHD_OPTION_END);

    }

Where memoryLimit is PAYLOAD_SIZE * 2 = 128MB.

Taking into account MHD documentation this seems to be a huge amount (https://www.gnu.org/software/libmicrohttpd/manual/libmicrohttpd.html):

The default is 32 kB (32*1024 bytes) as defined by the internal constant MHD_POOL_SIZE_DEFAULT. Values above 128k are unlikely to result in much benefit, as half of the memory will be typically used for IO, and TCP buffers are unlikely to support window sizes above 64k on most systems.

Thus, we have 3 order of magnitude above the maximum recomended value :)

Thus, the following modifications will be done:

• Use a CLI parameter to control the connection memory, e.g. -connectionMemory. Having this as a parameter (vs. hardwiring in the code) is good for future debug/tunning in unexpected situtations.
• Use a default for this parameter a value aligned with MHD documentation, i.e. 64Kb.

The connections memory footprint will be reduced a lot with this change.

[fgalan]

Max connection control and thread pool size implemented in PR https://github.com/telefonicaid/fiware-orion/issues/1477. Memory limit was implemented in PR https://github.com/telefonicaid/fiware-orion/pull/1425.

By the moment, it seems that MHD is tuneable enough, so this issue can be closed (if we need extra tunning, other -more specific- issues will be opened).