[KYUUBI #6754][AUTHZ] Improve the performance of Ranger access requests deduplication

[wankunde]

:mag: Description

Issue References 🔗

This pull request fixes #6754

Describe Your Solution 🔧

Right now in RuleAuthorization we use an ArrayBuffer to collect access requests, which is very slow because each new PrivilegeObject needs to be compared with all access requests.

Types of changes :bookmark:

• [x] Bugfix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request :coffin:

Add benchmark Before

Java HotSpot(TM) 64-Bit Server VM 17.0.12+8-LTS-286 on Mac OS X 14.6
Apple M3
Collecting files ranger access request:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
50000 files benchmark                            181863         189434         NaN         -0.0 -181863368958.0       1.0X

Behavior With This Pull Request :tada:

After

Java HotSpot(TM) 64-Bit Server VM 17.0.12+8-LTS-286 on Mac OS X 14.6
Apple M3
Collecting files ranger access request:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
50000 files benchmark                              1281           1310          33         -0.0 -1280563000.0       1.0X

Related Unit Tests

Exists UT

---

Checklist 📝

• [x] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (2d64255) to head (9d7d196). Report is 32 commits behind head on master.

Files with missing lines	Patch %	Lines
.../plugin/spark/authz/ranger/RuleAuthorization.scala	0.00%	4 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #6758 +/- ## ====================================== Coverage 0.00% 0.00% ====================================== Files 684 684 Lines 42282 42289 +7 Branches 5767 5769 +2 ====================================== - Misses 42282 42289 +7 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[bowenliang123]

Thanks for your contribution! Merged to master (1.10.0) .