search

astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs
https://hipscat-import.readthedocs.io
BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

Acquire raw DP0 data #240

Closed delucchi-cmu closed 6 months ago

delucchi-cmu commented 6 months ago

Get the raw data to test importing into hipscat. Ideally on epyc.

delucchi-cmu commented 6 months ago

The raw data has been downloaded to epyc, and converted to hipscat.

"/data3/epyc/data3/hipscat/catalogs/dp02"

image

I created a parquet metadata based off field ordering and additional key-value metadata from the dp02 schema. Passing this to the use_schema_file parameter on catalog import resolves the issue from https://github.com/astronomy-commons/hipscat-import/issues/239 (and is probably the only way to make sure that the created catalog contains the original key-value metadata).

I have more questions about where to go from here (and how to do it without having anyone be mad at me), but I think that this ticket is complete.