search

firebase / extensions

Source code for official Firebase extensions
https://firebase.google.com/products/extensions
Apache License 2.0
882 stars 373 forks source link

🐛 [firestore-bigquery-export] Compatibility with bigquery-emulator #2032

Closed simon-lejeune closed 2 months ago

simon-lejeune commented 2 months ago

[READ] Step 1: Are you in the right place?

Issues filed here should be about a feature request for a specific extension in this repository. To file a feature request that affects multiple extensions or the Firebase Extensions platform, please reach out to Firebase support directly.

[REQUIRED] Step 2: Extension name

This feature request is for extension: firestore-bigquery-export

What feature would you like to see?

Hello,

We are relying more and more on the firestore-bigquery-export in our codebase (17 instances to mirror our collections), however the development experience is not great. Indeed, Google does not provide an emulator for BigQuery, meaning we have to deploy to our staging environment to test the changes in our application (for example, we are also using the schema files to generate views from the _raw_changelog tables, then we have some sql of our own to aggregate them for some business logic. The latter is not testable locally.).

The following project is a pretty complete big query emualtor: bigquery-emulator. It is easy to start our node.js project using the firebase-admin library and point it to this emulator. With the official firebase auth and firestore emulators, we could run our setup locally. However, it doesn't seem to be possible to start locally the firestore-bigquery-export extension and told them to point to the big query emulator. This is the feature request of this issue.

I've stumbled upon this article where files are manually modified to allow connecting the extension to the emulator. Example: being able to pass the apiEndpoint as argument here. But it is of course not optimal to manually change the source code of libraries. Is this something you'd be willing to officially support?

How would you use it?

We'd like to be able to run the entire flow locally, both for the developer experience and also to implement automated tests (simple example: making a change in firestore, testing that the change is well in Big Query).

simon-lejeune commented 2 months ago

Closing while I actually do my research properly. It seems 1) I was mistaken saying it is easy to configure the BigQuery library to use the emulator, indeed changing the apiEndpoint is easy, but it will fail right after due to the emulator not being behind https and 2) I missed the BIGQUERY_EMULATOR_HOST env variable [0] that allows to workaround the previous issue.

Maybe what I want to do in this ticket is to pass this environment variable to the cloud functions of firestore-bigquery-export.

[0] https://github.com/googleapis/nodejs-bigquery/commit/8c54f20777a902a343035fcf09e63978d71135ad