search

astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs
https://hipscat-import.readthedocs.io
BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

Highest order when importing should not be needed #269

Closed nevencaplar closed 2 months ago

nevencaplar commented 6 months ago

https://github.com/astronomy-commons/hipscat-import/blob/9236dab7b045833b3f9af11362daaad2b21626da/src/hipscat_import/catalog/arguments.py#L63

The highest order should be determined from the imported data itself. It seems that now the import fails when importing high-density regions that should have higher orders and we do not specify the max-order, i.e., we use the default value that is too low.

delucchi-cmu commented 5 months ago

The highest order of the final hipscat catalog is determined by the data.

However, this argument is a runtime hint. If the user already knows that their catalog is smaller and will be partitioned a lower order (e.g. 5, 6, 7), setting this lower will speed up many of the import calculations. For larger catalogs, you'd need to set the value higher to ensure you're getting the best healpix resolution.

Depending on the implementation of https://github.com/astronomy-commons/hipscat-import/issues/154, we may be able to set this to a very high default value without much performance degradation.

nevencaplar commented 5 months ago

I think there are two problems here:

  1. I do believe (at least I wrote?!?) that the import fails if we use the value that is too low. That should not be the case
  2. If it is only a hint, it is not well named. Should go in the ticket I keep postponing about API changes.

I