🐛 [Stream Firestore to BigQuery] Update cloud function extension to gen 2

leslie-zi-pan commented 7 months ago

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

For general technical questions, post a question on StackOverflow with the firebase tag.
For general Firebase discussion, use the firebase-talk google group.
To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

Extension name: Stream Firestore to BigQuery
Extension version: 0.1.49

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

Failed extension deployment - specifically the firestore component of this extension. The extension allows database targetting (other than the default DB). However, deployment of this extension is still in 1st Gen Cloud function which limits the trigger to default database. This is contradictory to what the extension feature allows as there is a field to specify database ID. Please update to Gen 2. Thank you.

mikeymace commented 6 months ago

Hi!

Do we have a time frame for when the extensions will be upgraded to use the 2nd Gen Cloud Functions?

We are currently in the process of regionalising our Firestore databases (i.e. eur, us, shared) and we currently rely on the 'Stream Firestore to BigQuery' extension to get our data into BigQuery. Ideally, each database would be mapped to an equivalent (localised) dataset in BigQuery.

I have found the following behaviour for firebase/firestore-bigquery-export@0.1.49:

If I don't set the DATABASE_ID .env variable then the Cloud Function automatically defaults to 1st gen and DATABASE_ID = (default) see screenshot below.
If I do try and set DATABASE_ID = <location> in the .env then I get the following error message during deployment: ResourceErrorCode":"400","ResourceErrorMessage":"Cannot create Firestore trigger for .../databases/us-west2. Triggers must be created with a \"(default)\" Firestore Native database. See https://firebase.google.com/docs/firestore/extend-with-functions#limitations.

We also rely on npx firebaseextensions/fs-bq-import-collection for manual backfilling of existing Firestore data. Can you also confirm that this will be updated so that we can pass the database_id as an argument?

This is urgent from our side as our database migration is planned to go ahead early June and therefore any information on expected timeframes for this to be upgraded would be greatly appreciated otherwise unfortunately we will be forced to look for an alternative.

Thank you!

IchordeDionysos commented 6 months ago

@mikeymace I'd suggest looking for an alternative, e.g. copy the code and deploy it directly (aka not as Firebase Extension). This will take some time until it's fixed ...

pr-Mais commented 6 months ago

Hello everyone, tagging this as a feature request.

Unfortunately, it's not possible now to upgrade any extension to v2 Firestore triggers since extensions don't yet support them, once it does we will update the triggers.

For the contradiction between the params and actual functionality, I'm tracking this as an issue here #2090.

basvandorst commented 5 months ago

+1

general note: would be great if all (official) Firebase extensions also moving towards gen 2

adonistseriotis commented 3 weeks ago

Hello everyone, tagging this as a feature request.

Unfortunately, it's not possible now to upgrade any extension to v2 Firestore triggers since extensions don't yet support them, once it does we will update the triggers.

For the contradiction between the params and actual functionality, I'm tracking this as an issue here #2090.

I think somehow Algolia has implemented something similar on their Extension. Maybe, it is possible to do this here?

Default configuration is hardcoded here and even if this fails due to taskQueues implementation, the approach could be changed to match Algolia's sync approach. What do you think @pr-Mais ?

tlloydukdev commented 1 week ago

Default configuration is hardcoded here and even if this fails due to taskQueues implementation, the approach could be changed to match Algolia's sync approach. What do you think @pr-Mais ?

+1 from me

firebase / extensions