Apply `read_columns` kwarg in parquet reader - Githubissues

astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs

https://hipscat-import.readthedocs.io

BSD 3-Clause "New" or "Revised" License

5 stars 2 forks source link

Apply `read_columns` kwarg in parquet reader #315

Closed troyraen closed 1 month ago

troyraen commented 1 month ago

Change Description

Closes #296 Fixes a bug where ParquetReader.read takes a read_columns kwarg but doesn't actually use it.

[x] My PR includes a link to the issue that I am addressing

Solution Description

Pass the read_columns kwarg through to the read. This allows the mapping step to read only necessary columns (ra/dec) rather than all of them.
Add a column_names class property. This allows the user to import only a subset of columns.
Add unit test test_parquet_reader_columns.

Code Quality

[x] I have read the Contribution Guide and LINCC Frameworks Code of Conduct
[x] My code follows the code style of this project
[x] My code builds (or compiles) cleanly without any errors or warnings
[x] My code contains relevant comments and necessary documentation

Project-Specific Pull Request Checklists

Bug Fix Checklist

[x] My fix includes a new test that breaks as a result of the bug (if possible)
[ ] My change includes a breaking change
- [ ] My change includes backwards compatibility and deprecation warnings (if possible)

New Feature Checklist

[x] I have added or updated the docstrings associated with my feature using the NumPy docstring format
[ ] I have updated the tutorial to highlight my new feature (if appropriate)
[x] I have added unit/End-to-End (E2E) test cases to cover my new feature
[ ] My change includes a breaking change
- [ ] My change includes backwards compatibility and deprecation warnings (if possible)

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.53%. Comparing base (06b8379) to head (d4acbfe).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #315 +/- ## ======================================= Coverage 99.53% 99.53% ======================================= Files 25 25 Lines 1282 1284 +2 ======================================= + Hits 1276 1278 +2 Misses 6 6 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.