Handle remote URLs with query parameters

astronomy-commons / hipscat

Hierarchical Progressive Survey Catalog

https://hipscat.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

17 stars 3 forks source link

Handle remote URLs with query parameters #287

Closed Schwarzam closed 3 months ago

Schwarzam commented 3 months ago

Added functionality to recognize HTTP file systems and append query parameters to URLs in the pixel_catalog_files function. Introduced a helper function dict_to_query_urlparams to convert dictionaries to URL query strings.

Code Quality

[x] My code builds (or compiles) cleanly without any errors or warnings
[x] My code contains relevant comments and necessary documentation

New Feature Checklist

[x] I have added or updated the docstrings associated with my feature using the NumPy docstring format
[x] I have added unit/End-to-End (E2E) test cases to cover my new feature

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.78%. Comparing base (d57f59c) to head (fc0518b). Report is 42 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #287 +/- ## ========================================== + Coverage 93.68% 93.78% +0.10% ========================================== Files 57 57 Lines 1979 1995 +16 ========================================== + Hits 1854 1871 +17 + Misses 125 124 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Schwarzam commented 3 months ago

It seems that aiohttp module is needed just to check if path is HTTPFileSystem.

Is this really a good way?

github-actions[bot] commented 3 months ago

Before [6d427ac5]	After [b93abf25]	Ratio	Benchmark (Parameter)
75.6±0.3ms	76.9±0.2ms	1.02	benchmarks.MetadataSuite.time_load_partition_join_info
84.2±2ms	85.5±3ms	1.02	benchmarks.Suite.time_paths_creation
121±0.3ms	123±0.5ms	1.02	benchmarks.time_test_alignment_even_sky
76.1±0.4ms	76.9±0.5ms	1.01	benchmarks.MetadataSuite.time_load_partition_info_order7
361±2ms	364±3ms	1.01	benchmarks.Suite.time_outer_pixel_alignment
12.9±0.1ms	12.9±0.1ms	1	benchmarks.Suite.time_inner_pixel_alignment
39.7±1ms	39.5±1ms	1	benchmarks.Suite.time_pixel_tree_creation
984±4μs	988±8μs	1	benchmarks.time_test_cone_filter_multiple_order
19.2±0.4ms	19.0±0.4ms	0.99	benchmarks.MetadataSuite.time_load_partition_info_order6

Click here to view all benchmarks.