🐛 [firestore-bigquery-export] BigQuery sync table does not update if firestore collection is restored from backup

[EinarSnorrason]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

• For general technical questions, post a question on StackOverflow with the firebase tag.
• For general Firebase discussion, use the firebase-talk google group.
• To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export
• Extension version: 0.1.44
• Configuration values (redact info where appropriate):
  • Default configuration values for extension

[REQUIRED] Step 3: Describe the problem

Firestore has a feature to import and export data which is used to backup databases. If a collection that is streaming data to BigQuery imports using this feature, the changes are not synced to BigQuery. This means that using this feature can cause the tables to be permanently out of sync with each other.

Steps to reproduce:

• Create a collection and connect it to the Stream to BigQuery extension (default configuration values)
• Add a new document to the collection. It will be synced to BigQuery
• Use the Import/Export feature linked above to take a snapshot of the collection and store it to Cloud Storage
• Delete the document
• Import the snapshot

Expected result

The document gets recreated in firestore, and then synced to BigQuery so that the two tables have the same information.

Actual result

The document gets recreated in firestore, but not synced to BigQuery. The BigQuery table still says that the document has been deleted. The two tables are now out of sync.

[jauntybrain]

Hi @EinarSnorrason!

This extension does not support Firestore import events. It only handles onWrite events, which happen when a document in the specified collection is modified.

Since you are importing documents into BigQuery, have you considered using this instead?

[EinarSnorrason]

Thanks for the link, I'll take a look at that. I think I misunderstood the official description of the extension. It says:

If you create, update, delete, or import a document in the specified collection, this extension sends that update to BigQuery. You can then run queries on this mirrored dataset.

If the extension doesn't support Firestore import events I think this description should be modified to make it clearer what the extension can be used for.