fullstorydev / hauser

Service for moving your Fullstory export files to a data warehouse
MIT License
49 stars 23 forks source link

Preserve case in CustomVars JSON doc #30

Closed jameremo closed 6 years ago

jameremo commented 6 years ago

In previous PRs, we added case-insensitive matching of columns in the target warehouse with values from the incoming export bundle's JSON document. When we did that, we failed to preserve the case of values in the CustomVars JSON document, which is important for any downstream ETL processes. For example, let's say we originally were producing CustomVars that looked like:

{ "importantValue_str": "Important Data!!", "anImportantNum_real": 41 }

With the bug, the keys in CustomVars became all lowercase:

{ "importantvalue_str": "Important Data!!", "animportantnum_real": 41 }

This PR returns hauser to the state where case is preserved in CustomVars, while still doing case-insensitive matching to column names in the target warehouse.

jameremo commented 6 years ago

About making sure table columns are lowercase: The BQ implementation of Warehouse.GetExportTableColumns does forcibly make things lowercase. For Redshift, I read through these two docs:

https://docs.aws.amazon.com/redshift/latest/dg/r_names.html https://aws.amazon.com/about-aws/whats-new/2017/10/amazon-redshift-announces-support-for-uppercase-column-names/

There is a session-based parameter that you can optionally set to return column names in all uppercase. It doesn't appear from these docs that the Redshift practice of folding column names to lowercase is going to change any time soon, so I don't think it's urgent to add logic to force the column names to lowercase in hauser.

butanian commented 6 years ago

@jameremo good to know, thanks for digging into the documentation! 👍