Can healpix_29/hipscat_index be non-unique

astronomy-commons / hats

Hierarchical Adaptive Tiling Scheme

https://hats.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

17 stars 5 forks source link

Can healpix_29/hipscat_index be non-unique #345

Closed nevencaplar closed 1 month ago

nevencaplar commented 2 months ago

Investigate if there is anything that will break if healpix_29/hipscat_index is not a unique number, with a particular focus on pandas/DASK. One way to approach this would be to create a dataset with explicit healpix_29/hipscat_index that repeats and conduct a small analysis (e.g., periodogram) and see if there are failures.

smcguire-cmu commented 1 month ago

We've now added the non-unique index in the hats branch. There are some issues with pandas having a non-unique index. A few functions we used such as filter, and some loc calls didn't work, but it wasn't too bad to work around them. Our catalog functions shouldn't be affected by this, but users may run into similar issues if they do their own functions in map_partitions or other similar functions.