linkedin / openhouse

Open Control Plane for Tables in Data Lakehouse
https://www.openhousedb.org/
BSD 2-Clause "Simplified" License
273 stars 43 forks source link

Add storageType field to HouseTable Entity #81

Closed ctrezzo closed 1 month ago

ctrezzo commented 2 months ago

Summary

Issue Added a storageType field to the HouseTable Entity to support multiple storage types.

Resolves #80

Changes

For all the boxes checked, please include additional details of the changes made in this pull request.

Added storageType field to the HouseTable DTO.

Testing Done

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

For all the boxes checked, include additional details of the changes made in this pull request.

ctrezzo commented 2 months ago

Hey @HotSushi and @jainlavina! I have updated the pull request. Please take a look. Here are the main changes:

  1. storageType field is now populated with the HDFS default
  2. I added a unit test to validate defaults are set in the HouseTable builder properly
  3. I did a manual test with docker containers where I created a table via a spark shell and verified that the user_table_row table had the new row with "hdfs" set as the value for the storage_type column.
  4. All unit tests pass.
ctrezzo commented 2 months ago

@HotSushi Attached are the screenshots from the docker log during the manual test. Thanks!

Screenshot 2024-05-03 at 2 03 50 PM Screenshot 2024-05-03 at 2 03 23 PM Screenshot 2024-05-03 at 2 00 31 PM
ctrezzo commented 2 months ago

@HotSushi Attaching screenshot from HTS as well.

Screenshot 2024-05-06 at 12 46 02 PM
ctrezzo commented 2 months ago

Taking a look at the failed test