linkedin / openhouse

Open Control Plane for Tables in Data Lakehouse

https://www.openhousedb.org/

BSD 2-Clause "Simplified" License

273 stars 43 forks source link

JobScheduler switch using getAllTables to searchTables api #95

Closed jiang95-dev closed 1 month ago

jiang95-dev commented 2 months ago

Summary

JobScheduler is using getAllTables api which can bottlenecks the system and fails the jobs. This PR moves JobScheduler to use searchTables api.

Changes

[ ] Client-facing API Changes
[ ] Internal API Changes
[ ] Bug Fixes
[ ] New Features
[x] Performance Improvements
[ ] Code Style
[x] Refactoring
[ ] Documentation
[ ] Tests

Move all usage of getAllTables api inside TablesClient to searchTables api combined with getTable call.

Testing Done

[ ] Manually Tested on local docker setup. Please include commands ran, and their output.
[ ] Added new tests for the changes made.
[x] Updated existing tests to reflect the changes made.
[ ] No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
[ ] Some other form of testing like staging or soak time in production. Please explain.

Updated TablesClientTest to reflect the api change.

Additional Information

[ ] Breaking Changes
[ ] Deprecations
[ ] Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

autumnust commented 1 month ago

@jiang95-dev is this ready for review now ?