change date column to str prior to writing to parquet

ImagingDataCommons / idc-index-data

Python package providing the index to query and download data hosted by the NCI Imaging Data Commons

MIT License

1 stars 4 forks source link

change date column to str prior to writing to parquet #21

Closed vkt1414 closed 4 months ago

vkt1414 commented 4 months ago

Unlike csv, parquet stores schema info as well. Bigquery changes date columns db dtype column. This can cause inconsistencies for downstream operations like writing to pandas df.

Going forward, we may need to check if there are any date columns in upcoming indices (pathology, clinical, etc)