Closed adamkoch closed 5 months ago
Hey, thanks for raising this! I will look into it.
Apologies - I dug through the logs a bit more and the error was clearly logged and was for an instance that had partitions on. The error was:
ROW ERROR MESSAGE: Value 1553988622133000 for partition column created of the destination table
project:firestore_export.users_raw_changelog is outside the allowed bounds. You can only stream to partitions of date
range within 1825 days in the past and 366 days in the future relative to the current date. Integer value mapping to a
BigQuery timestamp column has to be microseconds since Unix Epoch time, please double check if your input data is
valid.
So the error is quite clear. I used the "creation" timestamp and some of the timestamps were > 5 years (1825) days.
[REQUIRED] Step 2: Describe your configuration
firestore-bigquery-export
0.1.50
asia-northeast1
false
NONE/disabled
100
(default)true
true
false
[REQUIRED] Step 3: Describe the problem
I set up the extension just yesterday so it's a fresh install and on the latest version. I set it up 5 times for 5 different document collections. The initial set up went fine and I could see my data showing up in BigQuery. I then used the script to backfill my data as well.
I see the following errors in my Cloud logs every few hours or so:
Note that this looks very similar to https://github.com/firebase/extensions/issues/235, although that issue seems old and is marked as closed and appears to be for earlier versions of the extension.
Steps to reproduce:
I'm not sure. I set up the script and it's all working as fine and now I see errors. I'm not sure how to troubleshoot further and see what data is actually causing this or if I can ignore it.
Expected result
No frequent errors.
Actual result
Frequent errors as per above.