cloudflare / cloudflare-gcp

Google Cloud Function to push json files from GC Storage to Big Query
Apache License 2.0
84 stars 49 forks source link

Custom RequestHeaders Field for Zero Trust has hyphens in it #102

Open dark1587 opened 2 years ago

dark1587 commented 2 years ago

The schema that's provided here does not include any headers used by Cloudflare Access (namely, RequestHeaders. This is known and addressed here.

However, I cannot add this to the BigQuery Schema easily, as the RequestHeaders isn't mentioned in the default schema, so I'm trying to hack around it by adding it manually inside the BigQuery Console.

This (I think) uses a RECORD data type, which requires nested fields to be added. But, FieldNames cannot have dashes (-) in them, only alphanumeric/underscore characters.

But the field I need to add to store usernames is cf-access-user (technically RequestHeaders.cf-access-user) has dashes... so in essence I cannot add in this field to BigQuery, unless I figure out some sort of magic to automatically convert the dashes into another name (to which, I'm having a heck of a time doing because I'm a newbie at BQ).

Any advice/insight you can provide would be extremely helpful.