astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs
https://hipscat-import.readthedocs.io
BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

Check the total number of rows throughout the catalog import pipeline #344

Closed delucchi-cmu closed 1 month ago

delucchi-cmu commented 1 month ago

Feature request

Allow users to specify the expected number of rows as an import argument. If specified, use this argument to check at various check-points in the pipeline that the number of rows being imported matches the expected number of rows.

Even if the user doesn't specify the expected number of rows, we should be able to determine the number of rows at the end of the mapping stage, and use this number throughout to ensure consistency.