search

cloudflare / cloudflare-gcp

Google Cloud Function to push json files from GC Storage to Big Query
Apache License 2.0
84 stars 49 forks source link

schema-gateway-dns.json causes errors when importing access gateway dns logs #111

Open dfloyd-blackcloak opened 2 years ago

dfloyd-blackcloak commented 2 years ago

The schema-gateway-dns.json file has incorrect values for QueryCategoryIDs (mode should be REPEATED) and RData needs to be nested as a RECORD type. The below changes fixed my errors:

Error 1: "Error while reading data, error message: JSON parsing error in row starting at position 0: Array specified for non-repeated field: QueryCategoryIDs."

Error 2: "Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: RData"

[ { "name": "ColoID", "type": "INTEGER", "mode": "NULLABLE" }, { "name": "ColoName", "type": "STRING", "mode": "NULLABLE" }, { "name": "Datetime", "type": "STRING", "mode": "NULLABLE" }, { "name": "DeviceID", "type": "STRING", "mode": "NULLABLE" }, { "name": "DstIP", "type": "STRING", "mode": "NULLABLE" }, { "name": "DstPort", "type": "INTEGER", "mode": "NULLABLE" }, { "name": "Email", "type": "STRING", "mode": "NULLABLE" }, { "name": "Location", "type": "STRING", "mode": "NULLABLE" }, { "name": "MatchedCategoryIDs", "type": "INTEGER", "mode": "NULLABLE" }, { "name": "Policy", "type": "STRING", "mode": "NULLABLE" }, { "name": "PolicyID", "type": "STRING", "mode": "NULLABLE" }, { "name": "Protocol", "type": "STRING", "mode": "NULLABLE" }, { "name": "QueryCategoryIDs", "type": "INTEGER", "mode": "REPEATED" }, { "name": "QueryNamestring", "type": "STRING", "mode": "NULLABLE" }, { "name": "QueryNameReversed", "type": "STRING", "mode": "NULLABLE" }, { "name": "QuerySize", "type": "INTEGER", "mode": "NULLABLE" }, { "name": "QueryType", "type": "STRING", "mode": "NULLABLE" }, { "name": "RData", "type": "RECORD", "mode": "REPEATED", "fields": [ { "name": "type", "type": "STRING", "mode": "NULLABLE" }, { "name": "data", "type": "STRING", "mode": "NULLABLE" } ] }, { "name": "ResolverDecision", "type": "STRING", "mode": "NULLABLE" }, { "name": "SrcIP", "type": "STRING", "mode": "NULLABLE" }, { "name": "SrcPort", "type": "INTEGER", "mode": "NULLABLE" }, { "name": "UserID", "type": "STRING", "mode": "NULLABLE" } ]