algolia / firestore-algolia-search

Apache License 2.0
112 stars 35 forks source link

Import script #96

Closed mariusg-shift closed 2 years ago

mariusg-shift commented 2 years ago

Hello, I'm having trouble with importing a firestore collection using the following script:

LOCATION=us-central1\ PROJECT_ID=hidden\ ALGOLIA_APP_ID=hidden\ ALGOLIA_API_KEY=hidden\ ALGOLIA_INDEX_NAME=test\ COLLECTION_PATH=activities\ FIELDS=name;title;description\ GOOGLE_APPLICATION_CREDENTIALS=</path/to/service/account/key>\ npx firestore-algolia-search

Terminal output: {"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"} {"algoliaAPIKey":"****","severity":"INFO","message":"Initializing extension with configuration"}

WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): y {"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read properties of undefined (reading 'split')\n at retrieveDataFromFirestore (/Users/marius/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:76:65)\n at /Users/marius/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:93:9\n at Interface._onLine (node:readline:485:5)\n at Interface._line (node:readline:864:8)\n at Interface._ttyWrite (node:readline:1216:14)\n at ReadStream.onkeypress (node:readline:288:10)\n at ReadStream.emit (node:events:520:28)\n at emitKeys (node:internal/readline/utils:358:14)\n at emitKeys.next ()\n at ReadStream.onData (node:internal/readline/emitKeypressEvents:61:36)"}

npm/npx version is 8.3.1

Rohithgilla12 commented 2 years ago

@mariusg-shift Yes, the script wasn't working I wrote a small custom Node JS script for this.

const algoliasearch = require("algoliasearch");
const dotenv = require("dotenv");
const firebase = require("firebase");

// load values from the .env file in this directory into process.env
dotenv.config();

// configure firebase
firebase.initializeApp({
  databaseURL: process.env.FIREBASE_DATABASE_URL,
  projectId: process.env.FIREBASE_APP_ID,
});
const firestore = firebase.firestore();

// configure algolia
const algolia = algoliasearch(
  process.env.ALGOLIA_APP_ID,
  process.env.ALGOLIA_API_KEY
);
const index = algolia.initIndex(process.env.ALGOLIA_INDEX_NAME);

var records = [];
firestore
  .collection("users")
  .get()
  .then((documentSnapshot) => {
    documentSnapshot.forEach((document) => {
      if (document.exists) {
        let user = document.data();
        user.objectID = document.id;
        records = records.concat(user);
      }
    });
    // Add or update new objects
    index
      .saveObjects(records)
      .then(() => {
        console.log("Users imported into Algolia");
      })
      .catch((error) => {
        console.error("Error when importing contact into Algolia", error);
        process.exit(1);
      });
  });

Maybe this could be helpful!

Rohithgilla12 commented 2 years ago

Also @maintainers if it helps, the error is something to do with the collection path, the function is trying to strip the text by "/" but this wouldn't be possible if user gives the base/root collection name users

mariusg-shift commented 2 years ago

@Rohithgilla12 thank you, the js script worked quite well. I was a bit hesitant to use something else than it is specified in the docs...

izanamiah commented 2 years ago

same error problem here, i wasn't able to back fill my index with my collection using this script. Any update on this issue?

smomin commented 2 years ago

Hello,

I am looking at this issue. I am seeing two problems here.

  1. The FIELDS should be comma separated, not semi-colon.
  2. If you're using UNIX flavored OS, please add \ at the end of each line for a multiline command or make the command a one liner if you don't want to use a \.

I hope that helps

izanamiah commented 2 years ago

Hi @smomin,

Thanks for your answer. I have all fields comma separated and added a \ at the end of each line but still have this problem.

smomin commented 2 years ago

@izanamiah can you provide details? it will help understand where the problem issue. Send me the command, error, screenshot or videos that will show me what you are seeing.

izanamiah commented 2 years ago

@smomin

the command i run was:

LOCATION=us-east1\
PROJECT_ID=hidden\
ALGOLIA_APP_ID=hidden\
ALGOLIA_API_KEY=hidden\
ALGOLIA_INDEX_NAME=hidden\
COLLECTION_PATH=hidden\
FIELDS=invNumber,name,country,city,year,type\
GOOGLE_APPLICATION_CREDENTIALS=path/to/firebase-adminsdk.json\

npx firestore-algolia-search

The error message i got was:

`{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"} {"algoliaAPIKey":"****","severity":"INFO","message":"Initializing extension with configuration"}

WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): `

after choose 'y'

{"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read properties of undefined (reading 'split')\n at retrieveDataFromFirestore (/Users/apple/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:76:65)\n at /Users/apple/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:93:9\n at Interface._onLine (node:readline:485:5)\n at Interface._line (node:readline:864:8)\n at Interface._ttyWrite (node:readline:1216:14)\n at ReadStream.onkeypress (node:readline:288:10)\n at ReadStream.emit (node:events:520:28)\n at emitKeys (node:internal/readline/utils:358:14)\n at emitKeys.next (<anonymous>)\n at ReadStream.onData (node:internal/readline/emitKeypressEvents:61:36)"}

The firebase service credential is newly generated. I double check the route is correct.

smomin commented 2 years ago

Have you tried it without the newline between the properties and the command.

LOCATION=us-east1\
PROJECT_ID=hidden\
ALGOLIA_APP_ID=hidden\
ALGOLIA_API_KEY=hidden\
ALGOLIA_INDEX_NAME=hidden\
COLLECTION_PATH=hidden\
FIELDS=invNumber,name,country,city,year,type\
GOOGLE_APPLICATION_CREDENTIALS=path/to/firebase-adminsdk.json\
npx firestore-algolia-search
izanamiah commented 2 years ago

@smomin I tried it as well, still the same error.

smomin commented 2 years ago

can you try the one liner?

LOCATION=us-east1 PROJECT_ID=hidden ALGOLIA_APP_ID=hidden ALGOLIA_API_KEY=hidden ALGOLIA_INDEX_NAME=hidden COLLECTION_PATH=hidden FIELDS=invNumber,name,country,city,year,type GOOGLE_APPLICATION_CREDENTIALS=path/to/firebase-adminsdk.json npx firestore-algolia-search

or

LOCATION=us-east1 PROJECT_ID=hidden ALGOLIA_APP_ID=hidden ALGOLIA_API_KEY=hidden ALGOLIA_INDEX_NAME=hidden COLLECTION_PATH=hidden FIELDS=invNumber,name,country,city,year,type GOOGLE_APPLICATION_CREDENTIALS=path/to/firebase-adminsdk.json npx firestore-algolia-search@0.5.9

Can you provide details on your environment? What OS, node, npm?

izanamiah commented 2 years ago

@smomin

I'm using the following environment: Mac OS 12.2.1, Node 16.14.0 NPM 8.3.1

The two one liner yield the error code below: {"severity":"WARNING","message":"[\n 'Payload size too big, skipping ...',\n Error: Record is too large.\n at Object.extract [as default] (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/firestore-algolia-search/lib/extract.js:60:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async processQuery (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/firestore-algolia-search/lib/import/index.js:54:29)\n]"} {"severity":"WARNING","message":"[\n 'Payload size too big, skipping ...',\n Error: Record is too large.\n at Object.extract [as default] (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/firestore-algolia-search/lib/extract.js:60:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async processQuery (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/firestore-algolia-search/lib/import/index.js:54:29)\n]"} {"severity":"INFO","message":"[ 'Sending rest of the Records to Algolia' ]"} {"severity":"INFO","message":"[ 'Preparing to send 183 record(s) to Algolia.' ]"} {"severity":"ERROR","message":"Error when performing Algolia index TypeError [ERR_HTTP_INVALID_HEADER_VALUE]: Invalid value \"undefined\" for header \"x-algolia-api-key\"\n at ClientRequest.setHeader (node:_http_outgoing:579:3)\n at new ClientRequest (node:_http_client:256:14)\n at Object.request (node:https:353:10)\n at /Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/@algolia/requester-node-http/dist/requester-node-http.cjs.js:33:72\n at new Promise (<anonymous>)\n at Object.send (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/@algolia/requester-node-http/dist/requester-node-http.cjs.js:18:20)\n at retry (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/@algolia/transporter/dist/transporter.cjs.js:222:38)\n at /Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/@algolia/transporter/dist/transporter.cjs.js:235:16\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5) {\n code: 'ERR_HTTP_INVALID_HEADER_VALUE'\n}"}

smomin commented 2 years ago

Ok, now this looks like an issue with the API key now.

Error when performing Algolia index TypeError [ERR_HTTP_INVALID_HEADER_VALUE]: Invalid value \"undefined\"

Can you check the Algolia logs?

smomin commented 2 years ago

@izanamiah is the issue resolved?

smomin commented 2 years ago

Closing this issue for now.

lupas commented 2 years ago

Had the same issue as the OP (I am on MacOS using the zsh shell) and landed here.

Putting it all on one line and running it in the zsh shell then worked 🥇 :

LOCATION="asia-southeast2" PROJECT_ID=HIDDEN ALGOLIA_APP_ID=HIDDEN ALGOLIA_API_KEY=HIDDEN ALGOLIA_INDEX_NAME=HIDDEN COLLECTION_PATH="FOO/FOO/FOO" FIELDS="id,basics.firstName,basics.lastName,basics.email" GOOGLE_APPLICATION_CREDENTIALS="/Users/ME/APP/credentials.json" npx firestore-algolia-search
345ml commented 2 years ago

https://github.com/algolia/firestore-algolia-search/issues/96#issuecomment-1117181625

Work for me

izanamiah commented 2 years ago

@smomin Sorry for my late response. It worked for me eventually. It turns out a problem of my text editor. What happened was when I pasted command from text editor into the terminal, it mess up with some format/char types. Thanks for helping.