feat(firestore-bigquery-export): allow firestore database selection

[jauntybrain]

Adds support for the multi-database feature recently introduced in Firestore. Allows users to specify a database ID for their Firestore instance.

Includes non-breaking dependency and code changes for firestore-bigquery-change-tracker.

Resolves #1743.

[pr-Mais]

@jauntybrain conflict

[pr-Mais]

can you add this new param to the _emulator's env file, that might be the reason tests are failing

[amungi]

Hi @jauntybrain / @pr-Mais : Can this feature for allowing firestore database selection be made available in the February release of the firestore-bigquery-export extension ? It would be really helpful for some of our projects where we are using multiple firestore databases in the same firebase project.

[amungi]

Hi @jauntybrain / @pr-Mais - is there any way to speed up the release of this feature?

[IchordeDionysos]

@pr-Mais any chance we could get this merged and uploaded to https://extensions.dev? ☺️ This is blocking us right now to setup some important analytics on data we added to a secondary database.

Given that support for multiple databases is now GA, this would be really helpful to have!

[pr-Mais]

Thanks all for your patience, we will release it ASAP.

[jauntybrain]

Hey all! Unfortunately, we have to pause the release of this feature for now. Extensions currently do not support v2 triggers that are required for named database selection (v1 trigger SDK does not support this feature).

We will be monitoring Firebase updates and implement this feature as soon as the v2 function support is available.

[IchordeDionysos]

Then why does the Firestore SDK support setting the database for Cloud Functions v1? 🤔

Okay that option was added 6 years ago...

[amungi]

@jauntybrain - some extensions, such as the Search Firestore with Algolia ( https://extensions.dev/extensions/algolia/firestore-algolia-search ) allow the Database ID to be configured, and we can specify Firestore database IDs other than the (default) database. How is it that some extensions have been able to do this without running into the v2 trigger limitations ?

[IchordeDionysos]

@jauntybrain well it seems like firestore-algolia-search does not properly support multiple databases. the databaseId is only taken into account for the reindexing operation but not the real-time updates:

https://github.com/search?q=repo%3Aalgolia%2Ffirestore-algolia-search%20databaseId&type=code https://github.com/algolia/firestore-algolia-search/blob/8976e65d4be4878b3ffcb9709a9725b38696da87/functions/src/index.ts#L44

For the real-time index operation the database is not configured: https://github.com/algolia/firestore-algolia-search/blob/8976e65d4be4878b3ffcb9709a9725b38696da87/functions/src/index.ts#L134-L136

[IchordeDionysos]

@jauntybrain @pr-Mais couldn't the extension use EventArc triggers and use Firestore as an event source? 🤔 https://firebase.google.com/docs/extensions/publishers/functions#customevents

This would work with Cloud Functions V2 and seems to allow using non-default databases: https://cloud.google.com/firestore/docs/extend-with-functions-2nd-gen#functions_cloudevent_firebase_firestore-nodejs

[jauntybrain]

Hi @IchordeDionysos, thank you for this suggestion. I'll review this approach to see if Eventarc triggers can be a suitable replacement for Firestore triggers until v2 support is added.

[jauntybrain]

Hi everyone. After extensive testing, it seems like this feature is working well after all. The Firestore trigger correctly recognizes doc changes from a secondary database, so no EventArc implementation is needed.

We will be releasing this feature in v0.1.46, as originally intended.

[amungi]

@jauntybrain - Thats great news !

[IchordeDionysos]

@jauntybrain any rough idea when you'd be able to push the release? ☺️

[vmalyi]

@jauntybrain, I'm curious what's the timeline for releasing the update containing this.

Thanks!

[alexregier]

Hello @jauntybrain , we just tested the update but can't get it working. When installing the extension we get an error Cannot create Firestore trigger for projects/XYZ/databases/ABC. Triggers must be created with a \"(default)\" Firestore Native database. See https://firebase.google.com/docs/firestore/extend-with-functions#limitations..

So it does not seem to work with v1 functions and that's also what's mentioned in the docs.

[jauntybrain]

Hi @alexregier, thanks for reporting this. Looking into it now.

[jauntybrain]

@alexregier I just tested version 0.1.48 and was able to successfully listen to changes in a secondary database instance with no errors. Could you please provide the full configuration of the extension with any sensitive information redacted?

[alexregier]

@jauntybrain I configured the extension via terraform, but this is the configuration to be used:

extension_ref                 = "firebase/firestore-bigquery-export"
extension_version             = "0.1.46"
params                        = {
   "BIGQUERY_PROJECT_ID"           = "XYZ"
   "COLLECTION_PATH"               = "someCollection/{someDoc}/anotherCollection"
   "DATABASE_ID"                   = "ABC"
   "DATASET_ID"                    = "abc"
   "DATASET_LOCATION"              = "eu"
   "DO_BACKFILL"                   = "yes"
   "TABLE_ID"                      = "abc"
   "TABLE_PARTITIONING"            = "MONTH"
   "TIME_PARTITIONING_FIELD"       = "timestamp"
   "TIME_PARTITIONING_FIELD_TYPE"  = "TIMESTAMP"
   "USE_NEW_SNAPSHOT_QUERY_SYNTAX" = "yes"
   "WILDCARD_IDS"                  = "true"
}

[neelance]

I'm seeing the same issue. Could it be because the named Firestore instance is not located in nam5?

[IchordeDionysos]

@neelance ours is also not in nam5.

We have gotten it to work somehow, we didn't make any huge changes (only updated from version 0.1.46 to 0.1.48)

I guess we also did the following:

1. Run terraform apply
2. Get error that non-default database is not supported
3. Run terraform plan
4. Run terraform apply (Then it worked)

[neelance]

@IchordeDionysos Are you sure that it actually worked? I also got a success message of the extension when I applied the config a second time, but the underlying Cloud Function was still in an error state and not working.

[IchordeDionysos]

@neelance ahh you are correct... Same for us ...

We were just scrambling with a different error to get BigQuery setup properly as there's an issue when you enable partitioning: https://github.com/firebase/extensions/pull/2028

[mikeymace]

@jauntybrain What is the status of this issue?

I have just tested 0.1.48 and 0.1.49 and experience the same as @IchordeDionysos and @neelance

I have also left a comment on this issue: https://github.com/firebase/extensions/issues/2072 (as I didn't see this issue earlier)