OctopusDeploy / Octodiff

| Public | 100% C# implementation of remote delta compression based on the rsync algorithm
Other
24 stars 87 forks source link

Reuse buffer to avoid memory spikes #39

Closed akirayamamoto closed 1 year ago

akirayamamoto commented 1 year ago

There is a breaking change in this PR. The behavior has not changed but one method needs more parameters. Data generated by the previous version is compatible with this version.

We improved the use of memory to avoid memory spikes and pod restarts in Octopus server.

For each chunk of data that we need to copy, instead of allocating a new array of bytes we will re-use the existing buffer.

Memory allocated before 102 GB. After 8 MB. See details below.

[sc-25897]

Before

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
ApplyBigDelta_Different 16.19 ms 0.311 ms 0.576 ms 375.0000 375.0000 375.0000 50 MB
ApplyBigDelta_Identical 2,066.69 ms 41.238 ms 98.802 ms 363000.0000 363000.0000 363000.0000 102400.77 MB
Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
ApplyBigDelta_Different 16.36 ms 0.279 ms 0.261 ms 375.0000 375.0000 375.0000 50 MB
ApplyBigDelta_Identical 1,983.46 ms 37.114 ms 93.113 ms 364000.0000 364000.0000 364000.0000 102400.77 MB

After

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
ApplyBigDelta_Different 4.577 ms 0.0874 ms 0.0935 ms 78.1250 78.1250 78.1250 8 MB
ApplyBigDelta_Identical 4.055 ms 0.0388 ms 0.0344 ms 62.5000 62.5000 62.5000 8 MB
Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
ApplyBigDelta_Different 4.319 ms 0.0842 ms 0.1475 ms 78.1250 78.1250 78.1250 8 MB
ApplyBigDelta_Identical 3.978 ms 0.0265 ms 0.0235 ms 62.5000 62.5000 62.5000 8 MB
shortcut-integration[bot] commented 1 year ago

This pull request has been linked to Shortcut Story #25897: Delta package upload could be causing memory spikes which might cause the pod to restart.