search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
3.12k stars 431 forks source link

Calling /hosts/{id} at high concurrency causes Fleet to consume too much RAM #23678

Open iansltx opened 1 week ago

iansltx commented 1 week ago

Fleet version: 4.58.0

💥  Actual behavior

This script of ours fires off a bunch of simultaneous requests to /hosts/{id}, causing RAM pressure on the server and potentially OOM issues if anything else is also causing RAM pressure. We're likely not the only ones wanting to use this endpoint at high concurrency.

🧑‍💻  Steps to reproduce

  1. Enroll a bunch of hosts
  2. Call GET /hosts/{id} on all of them simultaneously

🕯️ More info

Split from #22291 / #23078. TBD how much we can do about this, but maybe we can clean up incremental RAM usage per concurrent host pull enough to have documented guidance on how many simultaneous requests are acceptable per 4GB RAM instance. If we can get 200 concurrent requests without consuming 4GB of RAM that's probably sufficient?

🛠️ To fix

Find ways of reducing RAM footprint on this endpoint, and document how much RAM each concurrent execution of this endpoint takes.

sharon-fdm commented 4 days ago

Timebox 2 points for code review and attempt to make it lighter.