Schema check; check for correct columns and dtypes.

astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs

https://hipscat-import.readthedocs.io

BSD 3-Clause "New" or "Revised" License

5 stars 3 forks source link

Schema check; check for correct columns and dtypes. #374

Open nevencaplar opened 1 month ago

nevencaplar commented 1 month ago

This is part of verification pipeline tickets.

Check three places: ● _metadata ● _common_metadata ● random sample of Parquet file footers If starting with a Parquet or PyArrow schema, this can be as easy as pyarrow.schema.equals.

For HiPSCat, should also check the pandas metadata.