search

fullstorydev / hauser

Service for moving your Fullstory export files to a data warehouse
MIT License
48 stars 23 forks source link

Add support for schema changes to BigQuery #18

Closed butanian closed 6 years ago

butanian commented 6 years ago

This PR adds BigQuery support for schema changes. If hauser fields are not present in the existing export table in the specified dataset, we now add them and make sure all hauser fields are also present on any existing dataset table.

This is part (2) of a larger arc of work which will - (1) Add support for changing export schema on Redshift (2) Add support for changing export schema on BigQuery (3) Add new fields to the export

jameremo commented 6 years ago

cc @sanjaybv

butanian commented 6 years ago

I definitely want to combine logic between the redshift and bigquery implementations. Maybe even make a generic way that can be applied to other potential new implementations! There are some subtle differences between the two, you pointed one out, and the other is the Schema objects the two implementations use are different. I was hoping to discuss it some and then potentially do it in a different PR since it doesn't affect behavior and you know, how big a fan I am of granular PRs 😬