apache / celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.
https://celeborn.apache.org/
Apache License 2.0
862 stars 351 forks source link

[CELEBORN-1547] Worker#listTopDiskUseApps should return celeborn.metrics.app.topDiskUsage.count applications #2668

Closed SteNicholas closed 1 month ago

SteNicholas commented 1 month ago

What changes were proposed in this pull request?

Worker#listTopDiskUseApps should return celeborn.metrics.app.topDiskUsage.count applications.

Why are the changes needed?

Worker#listTopDiskUseApps returns 2x celeborn.metrics.app.topDiskUsage.count application, which is not same as celeborn.metrics.app.topDiskUsage.count configuration at present. Meanwhile, Worker#listTopDiskUseApps reuses StorageManager#topAppDiskUsage method to get the top application list used for estimated application disk usage. Therefore, Worker#listTopDiskUseApps should return celeborn.metrics.app.topDiskUsage.count applications.

$ curl http://celeborn-worker:9096/listTopDiskUsedApps|grep used|wc -l
100

Does this PR introduce any user-facing change?

No.

How was this patch tested?

$ curl http://celeborn-worker:9096/listTopDiskUsedApps|grep used|wc -l
50
SteNicholas commented 1 month ago

Ping @FMX, @AngersZhuuuu.

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 33.34%. Comparing base (ea6617c) to head (8e0b09a). Report is 13 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2668 +/- ## ========================================== - Coverage 39.83% 33.34% -6.49% ========================================== Files 239 310 +71 Lines 15026 18178 +3152 Branches 1362 1672 +310 ========================================== + Hits 5984 6059 +75 - Misses 8711 11780 +3069 - Partials 331 339 +8 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.