search

fullstorydev / hauser

Service for moving your Fullstory export files to a data warehouse
MIT License
49 stars 23 forks source link

Match csv column order to the export table schema #21

Closed butanian closed 6 years ago

butanian commented 6 years ago

This PR changes how we build the CSV that we use to populate the export table in the warehouses. Relevant changes are in commit 'fd7dbb9', the other commits are from this Bigquery PR which has not been merged in yet.

We have to do this in order to support bigquery table schemas where additional columns have been added since the previous hauser run. In bigquery there is no way to delete columns and there is no way to have the source file have a different column order than the table we are inserting to.

I was playing around with combining some of the code used between the Redshift and Bigquery, Warehouse implementations, but didn't like most the paths I was going down. Its probably my relative noob Golang status, so please do let me know if there are some things I can do better!

butanian commented 6 years ago

extracted the sub-steps from the load to warehouse function to be -> 1) upload File 2) defer delete file 3) actually load data

combined the flow for the process single file and process multiple files into one, so the loading workflow isn't duplicated.

butanian commented 6 years ago

@sanjaybv @jameremo PTAL