search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
3.11k stars 427 forks source link

Adding/Removing profiles in the UI takes a long time and times out with 30k hosts #21338

Open roperzh opened 2 months ago

roperzh commented 2 months ago

Fleet version: 4.55.0

Web browser and operating system:

💥  Actual behavior

Adding/removing a profile often times out with 30k hosts

This is a load testing result documented in here

🧑‍💻  Steps to reproduce

  1. Add a significant number of hosts with MDM turned on (doesn't need to be 30k to see the impact)
  2. Add at least 10 profiles
  3. Try to add/remove a profile, observe how the request takes longer than expected or times out

🕯️ More info

As noted in the load testing document, this happens when we set the profiles to "pending" (doesn't happen in the CLI)

Principal culprit seems to be:

https://github.com/fleetdm/fleet/blob/16d6757681a1e41f228eec798c4c0f0293b7cf0c/server/datastore/mysql/apple_mdm.go#L1777-L1782

gillespi314 commented 2 months ago

Heads up, this will likely be bigger than 2