search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
13.94k stars 3.53k forks source link

[fix][broker] Fix broker OOM when upload a large package. #22989

Closed shibd closed 1 day ago

shibd commented 5 days ago

Motivation

Using package API to upload a large package may cause broker OOM.

Because in the current implementation, will copy all data to the heap.

https://github.com/apache/pulsar/blob/2c6fcc7eb8343583ffb48dec937334a5f05afbae/pulsar-package-management/bookkeeper-storage/src/main/java/org/apache/pulsar/packages/management/storage/bookkeeper/DLOutputStream.java#L60-L65

Modifications

Each time, only copy the size of one record and then upload it directly.

There is no need to worry about batching, as the distributedlog client will handle the batching.

Verifying this change

Documentation

Matching PR in forked repository

PR in forked repository:

codecov-commenter commented 5 days ago

Codecov Report

Attention: Patch coverage is 85.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 73.43%. Comparing base (bbc6224) to head (f5b8da1). Report is 427 commits behind head on master.

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/apache/pulsar/pull/22989/graphs/tree.svg?width=650&height=150&src=pr&token=acYqCpsK9J&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/22989?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) ```diff @@ Coverage Diff @@ ## master #22989 +/- ## ============================================ - Coverage 73.57% 73.43% -0.15% - Complexity 32624 33266 +642 ============================================ Files 1877 1907 +30 Lines 139502 142951 +3449 Branches 15299 15578 +279 ============================================ + Hits 102638 104972 +2334 - Misses 28908 29946 +1038 - Partials 7956 8033 +77 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22989/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22989/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.52% <75.00%> (+2.93%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22989/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `24.73% <0.00%> (+0.40%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22989/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.45% <85.00%> (-0.39%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22989?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [.../management/storage/bookkeeper/DLOutputStream.java](https://app.codecov.io/gh/apache/pulsar/pull/22989?src=pr&el=tree&filepath=pulsar-package-management%2Fbookkeeper-storage%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fpackages%2Fmanagement%2Fstorage%2Fbookkeeper%2FDLOutputStream.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLXBhY2thZ2UtbWFuYWdlbWVudC9ib29ra2VlcGVyLXN0b3JhZ2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9wYWNrYWdlcy9tYW5hZ2VtZW50L3N0b3JhZ2UvYm9va2tlZXBlci9ETE91dHB1dFN0cmVhbS5qYXZh) | `90.62% <85.00%> (-0.29%)` | :arrow_down: | ... and [480 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22989/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)