🐛 [firestore-bigquery-export] fsimportexistingdocs enqueues in us-central1

[READ] Step 1: Are you in the right place?

✅

[REQUIRED] Step 2: Describe your configuration

Extension name: firestore-bigquery-export
Extension version: 0.1.45
Configuration values (redact info where appropriate):

    params = {
      COLLECTION_PATH            = "foo/*/bar"
      DATASET_ID                 = "my_dataset_id"
      DATASET_LOCATION           = "europe-west1"
      DO_BACKFILL                = "yes"
      IMPORT_COLLECTION_PATH     = "foo/*/bar"
      USE_COLLECTION_GROUP_QUERY = "yes"
      TABLE_ID                   = "foos"
      BIGQUERY_PROJECT_ID        = "my-project-id"
    }
    system_params = {
      "firebaseextensions.v1beta.function/location" = "europe-west1"
      "firebaseextensions.v1beta.v2function/location" = "europe-west1"
    }

[REQUIRED] Step 3: Describe the problem

I'm deploying using Terraform. As part of the deployment process, I've requested that it DO_BACKFILL. The extension seemingly installs its assets correctly (e.g., artifacts, cloud functions, cloud tasks, etc.) in the location I would expect: europe-west1

However, during the import process, the following errors start showing up in the log of the ext-firestore-to-bigquery-foos-fsimportexistingdocs Cloud Function:

ext-firestore-to-bigquery-campaigns-fsimportexistingdocsg7znuji01gql Unhandled error FirebaseFunctionsError: The principal (user or service account) lacks IAM permission "cloudtasks.tasks.create" for the resource "projects/my-project-id/locations/us-central1/queues/ext-firestore-to-bigquery-campaigns-fsimportexistingdocs" (or the resource may not exist).
    at FunctionsApiClient.toFirebaseError (/workspace/node_modules/firebase-admin/lib/functions/functions-api-client-internal.js:305:16)
    at FunctionsApiClient.enqueue (/workspace/node_modules/firebase-admin/lib/functions/functions-api-client-internal.js:146:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /workspace/lib/index.js:201:9
    at async /workspace/node_modules/firebase-functions/lib/common/providers/tasks.js:74:17 {
  errorInfo: {
    code: 'functions/permission-denied',
    message: 'The principal (user or service account) lacks IAM permission "cloudtasks.tasks.create" for the resource "projects/my-project-id/locations/us-central1/queues/ext-firestore-to-bigquery-campaigns-fsimportexistingdocs" (or the resource may not exist).'
  },
  codePrefix: 'functions'
}

For those who don't want to scroll to the right, here's the important bit: for the resource "projects/my-project-id/locations/us-central1/queues/ext-firestore-to-bigquery-campaigns-fsimportexistingdocs"

It appears that the Cloud Function is installed in europe-west1 but, when run, attempts to enqueue tasks in us-central1.

This was also observed in the final comment of the closed issue: https://github.com/firebase/extensions/issues/1844#issuecomment-1828330466

I believe the culprit is here: https://github.com/firebase/extensions/blob/a56408dcafac82abb2e34a8985be3aab5e059a18/firestore-bigquery-export/functions/src/index.ts#L255-L258

The documentation for taskQueue states that the first argument can be:

A fully qualified function resource name: projects/{project}/locations/{location}/functions/{functionName}

A partial resource name with location and function name, in which case the runtime project ID is used: locations/{location}/functions/{functionName}

A partial function name, in which case the runtime project ID and the default location, us-central1, is used: {functionName}

This line of code is using the third option, which forces it into us-central1.

You'll note that other calls to taskQueue in this same module specify the location: https://github.com/firebase/extensions/blob/next/firestore-bigquery-export/functions/src/index.ts#L192

If my hypothesis seems correct, I am happy to try my hand at fixing it in a PR.

Steps to reproduce:

Install the extension in a region different than us-central1 using a service account that has no access to us-central1 and enable the backfill-on-install option.

Expected result

My tables to be backfilled.

Actual result

Cloud Function errors (see above) and Cloud Task Queues that never complete.

firebase / extensions