search

ooni / probe

OONI Probe network measurement tool for detecting internet censorship
https://ooni.org/install
BSD 3-Clause "New" or "Revised" License
749 stars 142 forks source link

oohelperd: protect against overload #2649

Closed bassosimone closed 6 months ago

bassosimone commented 6 months ago

This issue is a follow-up of https://github.com/ooni/probe/issues/2413. Today we noticed excessive load in the oohelperd caused by spikes of synchronized requests hitting our oohelperd deployments.

We investigated the cause of the overload, which boils down to the following flame graph:

image

So, basically oohelperd is spending most of its time doing crypto for TLS handshakes. The impact on performance metrics is roughly like the following in terms of pretty much any metrics including time to service a request:

That is: resource consumption, time to service requests, CPU usage, etc. all go up pretty much in the same way.

We want to protect the oohelperd when there are too many clients by returning 504. Additionally, we want extra metrics to understand the time spent in each micro-operation (DNS, TCP, TLS, and HTTP).

bassosimone commented 6 months ago

Merged a fix and working on cutting 3.20.1 that will include the fix!