When generating a schema for the firebase bigquery export extension, Firebase Timestamp values do not show up in the latest schema but do in the changelog.
Looking in the code, there is a message that claims TIMESTAMP values can not be grouped, and are thus excluded. I feel like this may be a mistake. Here's the sample code in firestore-bigquery-export/scripts/gen-schema-view/src/schema.ts,
// For "latest" data views, certain types of fields cannot be used in
// "GROUP BY" clauses. We keep track of them so they can be explicitly
// transformed into groupable types later.
if (field.type === "array") {
arrays.push(qualifyFieldName(prefix, field.name));
}
if (field.type === "geopoint") {
geopoints.push(qualifyFieldName(prefix, field.name));
}
if (field.type === "timestamp") {
timestamps.push(qualifyFieldName(prefix, field.name));
}
I don't understand why this applies to timestamps (it's very possible I simply am ignorant on the subject -- I'm no SQL expert). When I comment out the timestamp clause, the schemas and resulting views work exactly as expected, with SQL Timestamp values where I had registered a timestamp in my schema.
Steps to reproduce:
Create a firestore collection with a timestamp field.
Create a schema.json file for this collection and register the timestamp field.
Run the generate-schema-view script
Notice the timestamp does not show up when you execute a select statement on the resulting _latest view.
My "fix"
Simply remove the if statement schema.ts that adds timestamp values to the exclusion array.
Hey @Neilpoulin, thanks for raising the issue. You're correct, I'll be making a PR that will allow the timestamp to come through in the appropriate column that was mapped in the schema for the "latest" View.
[REQUIRED] Step 2: Describe your configuration
[REQUIRED] Step 3: Describe the problem
When generating a schema for the firebase bigquery export extension, Firebase Timestamp values do not show up in the
latest
schema but do in the changelog.Looking in the code, there is a message that claims
TIMESTAMP
values can not be grouped, and are thus excluded. I feel like this may be a mistake. Here's the sample code infirestore-bigquery-export/scripts/gen-schema-view/src/schema.ts
,I don't understand why this applies to timestamps (it's very possible I simply am ignorant on the subject -- I'm no SQL expert). When I comment out the
timestamp
clause, the schemas and resulting views work exactly as expected, with SQL Timestamp values where I had registered a timestamp in my schema.Steps to reproduce:
_latest
view.My "fix" Simply remove the if statement
schema.ts
that adds timestamp values to the exclusion array.