pipe-cd / pipecd

The One CD for All {applications, platforms, operations}
https://pipecd.dev
Apache License 2.0
1.06k stars 149 forks source link

Make controller's os.RemoveAll single-threaded #5217

Closed Warashi closed 5 days ago

Warashi commented 1 week ago

What this PR does / why we need it:

This PR modifies the planner/scheduler cleanup to be single-threaded in order to reduce the file IO loads. I did a load test with the following scenario and discovered that os.RemoveAll creates a high load for piped:

In this scenario, the piped requires about 20 minutes to complete all deployments. After this PR, the piped requires about 15 minutes to complete.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?: Yes

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 0% with 18 lines in your changes missing coverage. Please review.

Project coverage is 22.81%. Comparing base (5b9d417) to head (efa4e6d). Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
pkg/app/piped/controller/controller.go 0.00% 18 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #5217 +/- ## ========================================== - Coverage 22.83% 22.81% -0.03% ========================================== Files 420 420 Lines 45302 45316 +14 ========================================== - Hits 10344 10338 -6 - Misses 34163 34183 +20 Partials 795 795 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.