sul-dlss / dlme-airflow

This is a new repository to capture the work related to the DLME ETL Pipeline and establish airflow
Apache License 2.0
1 stars 0 forks source link

Prevent pandas data type inference #484

Closed edsu closed 4 months ago

edsu commented 4 months ago

When reading JSON data back into Pandas we have to be careful not to let Pandas infer types or else a column named 'date' with a string '1960' in it will get parsed as a datetime, which then causes it to be persisted incorrectly.

I added a test that initially failed, until I made the correction to how JSON is read in.

Fixes #482