search

Chicago / opengrid

A user-friendly, map-based tool to combine and explore real-time or historical data.
http://opengrid.io
Other
247 stars 53 forks source link

Detect and deal with changes in structure to current datasets #292

Closed levyj closed 7 years ago

levyj commented 7 years ago

From time to time, the structure of a dataset will change. How can Open Grid find out about and deal with these changes? This is related to the same issue in Plenario -- https://github.com/UrbanCCD-UChicago/plenario/issues/291.

tomschenkjr commented 7 years ago

@heyzoos - you mentioned this was deployed on Plenario now, right?

HeyZoos commented 7 years ago

@tomschenkjr - Yep, during the ETL process Plenario completely replaces the existing dataset instead of trying to perform upserts. This has been enough to ingest datasets where the number of columns changes.

HeyZoos commented 7 years ago

@tomschenkjr - However this does not address communicating the changes to Open Grid, the metadata about the dataset will change but that would have to actively be checked for.

tomschenkjr commented 7 years ago

@HeyZoos - great, and that does happen so closing the issue.