firestore-bigquery-export failing

[madmacc]

• Extension name: firestore-bigquery-export

I have installed the Export Collections to BigQuery extension and it works fine however now I am attempting to import my existing data using firestore-bigquery-export by following the instructions here: https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md

However I am getting the error detailed below. Note: I get exactly the same error even if I enter a firebase projectid or collection that does not exist.

This may be an local environment issue, if so and this is not an appropriate place for this issue please let me know and close it.

$ node -v v12.13.0

$ npm -v 6.12.0

Steps to reproduce:

Run: npx @firebaseextensions/fs-bq-import-collection Enter the ${PROJECT_ID}, ${COLLECTION_PATH} and ${DATASET_ID} parameters as follows: my-fb-project-id, members, qmtbc_dev_export

Expected result

Collection imports successfully into BigQuery

Actual result

Error returned:

Importing data from Cloud Firestore Collection: members, to BigQuery Dataset: qmtbc_dev_export, Table: members_raw_changelog
Error unlinking journal file C:\Users\fullo\AppData\Roaming\npm-cache\_npx\10884\node_modules\@firebaseextensions\fs-bq-import-collection\lib/from-members-to-qmtbc-dev:qmtbc_dev_export:members_raw_changelog
after successful import: Error: ENOENT: no such file or directory, unlink 'C:\Users\fullo\AppData\Roaming\npm-cache\_npx\10884\node_modules\@firebaseextensions\fs-bq-import-collection\lib\from-members-to-qmtbc-dev:qmtbc_dev_export:members_raw_changelog'
---------------------------------------------------------
Finished importing 0 Firestore rows to BigQuery
---------------------------------------------------------
Error importing Collection to BigQuery: Error: EPERM: operation not permitted, unlink 'C:\Users\fullo\AppData\Roaming\npm-cache\_npx\10884\node_modules\@firebaseextensions\fs-bq-import-collection\node_modules\grpc\src\node\extension_binary\node-v64-win32-x64-unknown\grpc_node.node'

Also I looked at the script here: https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/scripts/import/src/index.ts

And the error is being generated at line 187. It looks like it may throw this error if no data is found or if can't access the project/collection.

[Ehesp]

Looks like a Windows permission issue. I'll get a local Window environment running and give it a go.

[madmacc]

@Ehesp Much appreciated. It would be good to know if it actually works so I can continue to try and resolve. I have 2 other computers including a Mac that I can try on but will take a bit of time setting up.

Among other things I tried: node v8 and v10 (currently on v10). I tried using task manager to make sure no other versions on node were running and locking it. I deleted and reinstalled all my npm packages several times.

[madmacc]

Any update on this? I would like to know if it is working for others using Windows. If so then I can justify spending some more time trying to get it working.

[chenlevy]

Same error here, any workaround?

[madmacc]

@chenlevy Not yet. I am still waiting to see if anyone has an answer. Others have said it works on a Mac so I may try that.

[chenlevy]

@madi8229 I think I figured it out, the problem is with colons in the cursor pointer file name mentioned in the script. Windows file names shouldn't contain colons. I've changed it to underscores and run it locally and it worked perfectly.

131 Also opened a pull request

[Ehesp]

Fixed via https://github.com/firebase/extensions/pull/131

[iDeliverApps]

I am still having this issue with Mac. No problem uploading schema files and having data populate against those, but I can't get any other result for import than Error unlinking journal file '/Users/.... ... Error: ENOENT: no such file or directory, unlink '/Users/....

`? What is your Firebase project ID? ideliver-cloud
? What is the path of the the Cloud Firestore Collection you would like to import from? (This may, or may not, be the same Collection for which you p
lan to mirror changes.) idriver-app\events\incoming
? What is the ID of the BigQuery dataset that you would like to use? (A dataset will be created if it doesn't already exist) ideliver_app
? What is the identifying prefix of the BigQuery table that you would like to import to? (A table will be created if one doesn't already exist) incom
ing
? How many documents should the import stream into BigQuery at once? 300
Importing data from Cloud Firestore Collection: idriver-app\events\incoming, to BigQuery Dataset: ideliver_app, Table: incoming_raw_changelog
Error unlinking journal file /Users/tim/.npm/_npx/4672/lib/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-idriver-app\events\incoming-to-ideliver-cloud_ideliver_app_incoming_raw_changelog after successful import: Error: ENOENT: no such file or directory, unlink '/Users/tim/.npm/_npx/4672/lib/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-idriver-app\events\incoming-to-ideliver-cloud_ideliver_app_incoming_raw_changelog'
---------------------------------------------------------
Finished importing 0 Firestore rows to BigQuery
---------------------------------------------------------`

Any help would be greatly appreciated

[mustafaekim]

I am having the same error on a Windows machine

[Gibbo3771]

Hello from 2021, I am sad to say this is still broken on MacOS. Exact same error.

[dackers86]

Hi @Gibbo3771.

Can you let me know the following is related? https://github.com/firebase/extensions/pull/589.

We are looking at further improvements for handling file paths for the export functions

[Gibbo3771]

I will have a look but I have also noticed there is a guard here that checks if the cursor file exists before doing anything, but further down (where it unlinks) right here there is no guard.

So the error people are most likely seeing is because the import was successful (and most likely 0 records) and this line is trying to unlink a file that simply doesn't exist, and never will (because 0 records).

I think I have a problem with my gcloud credentials or IAM role that is preventing it from importing the data, and this error has just blinded me to that fact.

EDIT: I am aware there is a try/catch there, but spitting out the entire error right after printing a "success" message in a catch block (?) is a bit weird.

[dackers86]

Ok, in that case I'll re-open for investigation.

[Gibbo3771]

@dackers86 to answer your queston, it looks to be unrelated.

I think I have a permission issue and there is no error handling for it, I would have thought the google sdk would throw if you tried to do something without the correct IAM roles, and there is no try/catch wrapped around the write call to bigquery, shown here.

So right now I have zero clue why my import is reporting 0 records.

[dackers86]

Hi @Gibbo3771

I've been so far unable to recreate. We are looking to release better file sanitizing for paths as mentioned earlier.

Could this be a potential test post release? https://github.com/firebase/extensions/pull/589

[dackers86]

Closing until further feedback provided. The above PR could be a potential fix for this.

[honmaaax]

I fixed it with the command below. But be careful if you do it.

unset FIRESTORE_EMULATOR_HOST
unset GCLOUD_PROJECT

[ChiamaShawn]

In my case on Mac OS setting permissions for current user on the .npm/_npx folder helped

sudo chown -R USER /Users/USER/.npm/_npx/ 

Replace USER with your local username

[cabljac]

I got this issue on a Mac, when i had GOOGLE_APPLICATION_CREDENTIALS set to something.

I managed to fix it by doing unset GOOGLE_APPLICATION_CREDENTIALS

Still not clear to me what's going on...

[cabljac]

If anyone is getting this issue and is unable to fix it with the above techniques, please reply to this thread and i will reopen. Otherwise I think we could do with better error handling in this situation perhaps.

[matheusleo]

Hey @cabljac!

I am currently having a issue to import export the data! I tried the techniques above, but I could not solve my problem.

I have all the permissions required - I am the owner of the project. But, I have not found any solution that works to me

My OS is an Ubuntu 22.10 and usually I do not have this kind of issue

[pehrlich-arbor]

Getting this as well - repro'd on two different OSX machines (14.1.2). Node v18.19.0.

However, it also doesn't seem to matter. My data is getting copied just fine. I just had to use the right syntax to get a a nested collection with group queries.