It would be great if hipscat-import could infer data types from ECSV catalogs, like Gaia DR3. Currently, user is required to provide an empty parquet file with a schema for this catalog.
astropy could help with this task. While its ECSV parser looks very slow, we still could use it just for data type inference:
from astropy.io import ascii
empty_astropy_table = ascii.read(gaia_file, format='ecsv', data_end=1)
empty_astropy_table.write('schema.parquet')
Before submitting
Please check the following:
[x] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
[x] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
[x] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.
Feature request
It would be great if
hipscat-importcould infer data types from ECSV catalogs, like Gaia DR3. Currently, user is required to provide an empty parquet file with a schema for this catalog.astropycould help with this task. While its ECSV parser looks very slow, we still could use it just for data type inference:Before submitting Please check the following: