🐛 [firestore-bigquery-export] Task size too large errors occuring even with EXCLUDE_OLD_DATA set to yes/true

[747project]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

• For general technical questions, post a question on StackOverflow with the firebase tag.
• For general Firebase discussion, use the firebase-talk google group.
• To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export
• Extension version: 0.1.50
• Configuration values (redact info where appropriate):
  • BigQuery Dataset location: us
  • BigQuery Project ID: xxx
  • Database ID: (default)
  • Collection path: xxx
  • Enable Wildcard Column field with Parent Firestore Document IDs (Optional): false
  • Dataset ID: xxx
  • Table ID: xxx
  • BigQuery SQL table Time Partitioning option type (Optional): DAY
  • BigQuery Time Partitioning column name (Optional): timestamp
  • Firestore Document field name for BigQuery SQL Time Partitioning field option (Optional): Parameter not set
  • BigQuery SQL Time Partitioning table schema field(column) type (Optional): omit
  • BigQuery SQL table clustering (Optional): document_id
  • Maximum number of synced documents per second (Optional): 100
  • Backup Collection Name (Optional): Parameter not set
  • Transform function URL (Optional): Parameter not set
  • Use new query syntax for snapshots: no
  • Exclude old data payloads (Optional): yes
  • Use Collection Group query (Optional): no
  • Cloud KMS key name (Optional): Parameter not set

[REQUIRED] Step 3: Describe the problem

Even when using the EXCLUDE_OLD_DATA setting to prevent old_data from being populated, we are still seeing Task size too large errors on many messages. This might mean that firestore payloads close to 1MB are being padded in a way that the subsequent Task exceeds 1MB

Steps to reproduce:

1. Install extension version 0.1.50, ensure that Exclude old data payloads is set to yes
2. Write a large document to firestore

Expected result

No Task size too large errors should appear at all.

Actual result

Observing many Task size too large errors in logs.