search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
3.03k stars 420 forks source link

Large software installers can run Fleet out of memory. #20595

Open rfairburn opened 2 months ago

rfairburn commented 2 months ago

Fleet version: 4.54.0

Web browser and operating system: N/A

💥  Actual behavior

With the change of the upload window from 2 minutes to 4 for software installers, it is now possible for someone with at least a gigabit connection to upload a 4 GB file within the upload window. Since we recommend that Fleet be configured with 4 GB of ram, this will cause the server to crash if we were to increase the limit above 500MB in the future.

🧑‍💻  Steps to reproduce

  1. Have fast internet.
  2. Upload ~4GB or larger package with the limit removed.
  3. Watch fleet container crash (502 error)

🕯️ More info (optional)

During testing in the QAWolf env, when it had less than 4GB allocated containers would run out of memory during a 400mb software installer upload on a 1GB configuration.

In some environments software installer packages can be very large. Adobe packages can be 20-25GB.

Possible solution might be to stream upload directly to S3 (handing the request body to a io.writer).

Another would be to inspect inside the .pkg like this app: https://www.mothersruin.com/software/SuspiciousPackage/

This would probably be in some "pending metadata extraction state", then kick off an async task after the upload was done to do the inspection

mostlikelee commented 3 weeks ago

Microsoft Office for Mac pkg ~= 2.5GB

lukeheath commented 3 weeks ago

@rfairburn Thanks for filing this!

@georgekarrv Given that this has the potential to crash Fleet, I am prioritizing to the drafting board for estimation.

rfairburn commented 3 weeks ago

I think our size limitations for max uploads might be ok for 4gb of memory currently in many cases, but I've seen 4gb of memory occasionally not be enough in the real world without the installers coming into play at all. I'm concerned in particular with instances that both have a large number of hosts and a near-max-allowed-size software installer.

georgekarrv commented 1 week ago

Hey team! Please add your planning poker estimate with Zenhub @dantecatalfamo @gillespi314 @jahzielv @mna @roperzh

georgekarrv commented 1 week ago

Please add your planning poker estimate with Zenhub @ghernandez345

noahtalerman commented 3 days ago

Hey @lukeheath heads up, this user story didn't make it into the upcoming engineering sprint due to capacity.

It's still prioritized. We left it on the drafting board so that it can be pulled into the next engineering sprint.