astronomy-commons / hipscat

Hierarchical Progressive Survey Catalog
https://hipscat.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
14 stars 3 forks source link

Support list of paths for parquet dataset. #288

Closed delucchi-cmu closed 3 weeks ago

delucchi-cmu commented 3 weeks ago

Change Description

This is needed for the implementation of https://github.com/astronomy-commons/hipscat-import/issues/308.

We read a parquet dataset based on a list of input files. The method inside hipscat is already well-suited for interpreting storage_options and non-local file paths. This adds support for list-like input sources.

Code Quality

codecov[bot] commented 3 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.69%. Comparing base (6d427ac) to head (eba7153).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #288 +/- ## ========================================== + Coverage 93.68% 93.69% +0.01% ========================================== Files 57 57 Lines 1979 1983 +4 ========================================== + Hits 1854 1858 +4 Misses 125 125 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

github-actions[bot] commented 3 weeks ago
Before [6d427ac5] After [7c311bf8] Ratio Benchmark (Parameter)
18.2±0.5ms 19.3±0.6ms 1.06 benchmarks.MetadataSuite.time_load_partition_info_order6
990±3μs 1.02±0ms 1.03 benchmarks.time_test_cone_filter_multiple_order
75.6±0.4ms 76.0±0.3ms 1.01 benchmarks.MetadataSuite.time_load_partition_info_order7
122±1ms 123±1ms 1.01 benchmarks.time_test_alignment_even_sky
75.7±0.5ms 76.0±0.3ms 1 benchmarks.MetadataSuite.time_load_partition_join_info
12.7±0.2ms 12.7±0.2ms 1 benchmarks.Suite.time_inner_pixel_alignment
84.4±2ms 84.0±2ms 1 benchmarks.Suite.time_paths_creation
365±2ms 362±3ms 0.99 benchmarks.Suite.time_outer_pixel_alignment
41.3±1ms 39.6±1ms 0.96 benchmarks.Suite.time_pixel_tree_creation

Click here to view all benchmarks.