Closed mariusg-shift closed 2 years ago
@mariusg-shift Yes, the script wasn't working I wrote a small custom Node JS script for this.
const algoliasearch = require("algoliasearch");
const dotenv = require("dotenv");
const firebase = require("firebase");
// load values from the .env file in this directory into process.env
dotenv.config();
// configure firebase
firebase.initializeApp({
databaseURL: process.env.FIREBASE_DATABASE_URL,
projectId: process.env.FIREBASE_APP_ID,
});
const firestore = firebase.firestore();
// configure algolia
const algolia = algoliasearch(
process.env.ALGOLIA_APP_ID,
process.env.ALGOLIA_API_KEY
);
const index = algolia.initIndex(process.env.ALGOLIA_INDEX_NAME);
var records = [];
firestore
.collection("users")
.get()
.then((documentSnapshot) => {
documentSnapshot.forEach((document) => {
if (document.exists) {
let user = document.data();
user.objectID = document.id;
records = records.concat(user);
}
});
// Add or update new objects
index
.saveObjects(records)
.then(() => {
console.log("Users imported into Algolia");
})
.catch((error) => {
console.error("Error when importing contact into Algolia", error);
process.exit(1);
});
});
Maybe this could be helpful!
Also @maintainers if it helps, the error is something to do with the collection path, the function is trying to strip the text by "/" but this wouldn't be possible if user gives the base/root collection name users
@Rohithgilla12 thank you, the js script worked quite well. I was a bit hesitant to use something else than it is specified in the docs...
same error problem here, i wasn't able to back fill my index with my collection using this script. Any update on this issue?
Hello,
I am looking at this issue. I am seeing two problems here.
FIELDS
should be comma separated, not semi-colon.\
at the end of each line for a multiline command or make the command a one liner if you don't want to use a \
.I hope that helps
Hi @smomin,
Thanks for your answer. I have all fields comma separated and added a \ at the end of each line but still have this problem.
@izanamiah can you provide details? it will help understand where the problem issue. Send me the command, error, screenshot or videos that will show me what you are seeing.
@smomin
the command i run was:
LOCATION=us-east1\
PROJECT_ID=hidden\
ALGOLIA_APP_ID=hidden\
ALGOLIA_API_KEY=hidden\
ALGOLIA_INDEX_NAME=hidden\
COLLECTION_PATH=hidden\
FIELDS=invNumber,name,country,city,year,type\
GOOGLE_APPLICATION_CREDENTIALS=path/to/firebase-adminsdk.json\
npx firestore-algolia-search
The error message i got was:
`{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"} {"algoliaAPIKey":"****","severity":"INFO","message":"Initializing extension with configuration"}
WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): `
after choose 'y'
{"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read properties of undefined (reading 'split')\n at retrieveDataFromFirestore (/Users/apple/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:76:65)\n at /Users/apple/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:93:9\n at Interface._onLine (node:readline:485:5)\n at Interface._line (node:readline:864:8)\n at Interface._ttyWrite (node:readline:1216:14)\n at ReadStream.onkeypress (node:readline:288:10)\n at ReadStream.emit (node:events:520:28)\n at emitKeys (node:internal/readline/utils:358:14)\n at emitKeys.next (<anonymous>)\n at ReadStream.onData (node:internal/readline/emitKeypressEvents:61:36)"}
The firebase service credential is newly generated. I double check the route is correct.
Have you tried it without the newline between the properties and the command.
LOCATION=us-east1\
PROJECT_ID=hidden\
ALGOLIA_APP_ID=hidden\
ALGOLIA_API_KEY=hidden\
ALGOLIA_INDEX_NAME=hidden\
COLLECTION_PATH=hidden\
FIELDS=invNumber,name,country,city,year,type\
GOOGLE_APPLICATION_CREDENTIALS=path/to/firebase-adminsdk.json\
npx firestore-algolia-search
@smomin I tried it as well, still the same error.
can you try the one liner?
LOCATION=us-east1 PROJECT_ID=hidden ALGOLIA_APP_ID=hidden ALGOLIA_API_KEY=hidden ALGOLIA_INDEX_NAME=hidden COLLECTION_PATH=hidden FIELDS=invNumber,name,country,city,year,type GOOGLE_APPLICATION_CREDENTIALS=path/to/firebase-adminsdk.json npx firestore-algolia-search
or
LOCATION=us-east1 PROJECT_ID=hidden ALGOLIA_APP_ID=hidden ALGOLIA_API_KEY=hidden ALGOLIA_INDEX_NAME=hidden COLLECTION_PATH=hidden FIELDS=invNumber,name,country,city,year,type GOOGLE_APPLICATION_CREDENTIALS=path/to/firebase-adminsdk.json npx firestore-algolia-search@0.5.9
Can you provide details on your environment? What OS, node, npm?
@smomin
I'm using the following environment: Mac OS 12.2.1, Node 16.14.0 NPM 8.3.1
The two one liner yield the error code below:
{"severity":"WARNING","message":"[\n 'Payload size too big, skipping ...',\n Error: Record is too large.\n at Object.extract [as default] (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/firestore-algolia-search/lib/extract.js:60:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async processQuery (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/firestore-algolia-search/lib/import/index.js:54:29)\n]"} {"severity":"WARNING","message":"[\n 'Payload size too big, skipping ...',\n Error: Record is too large.\n at Object.extract [as default] (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/firestore-algolia-search/lib/extract.js:60:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async processQuery (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/firestore-algolia-search/lib/import/index.js:54:29)\n]"} {"severity":"INFO","message":"[ 'Sending rest of the Records to Algolia' ]"} {"severity":"INFO","message":"[ 'Preparing to send 183 record(s) to Algolia.' ]"} {"severity":"ERROR","message":"Error when performing Algolia index TypeError [ERR_HTTP_INVALID_HEADER_VALUE]: Invalid value \"undefined\" for header \"x-algolia-api-key\"\n at ClientRequest.setHeader (node:_http_outgoing:579:3)\n at new ClientRequest (node:_http_client:256:14)\n at Object.request (node:https:353:10)\n at /Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/@algolia/requester-node-http/dist/requester-node-http.cjs.js:33:72\n at new Promise (<anonymous>)\n at Object.send (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/@algolia/requester-node-http/dist/requester-node-http.cjs.js:18:20)\n at retry (/Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/@algolia/transporter/dist/transporter.cjs.js:222:38)\n at /Users/izanamiah/.npm/_npx/da8f72bd08bc95ff/node_modules/@algolia/transporter/dist/transporter.cjs.js:235:16\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5) {\n code: 'ERR_HTTP_INVALID_HEADER_VALUE'\n}"}
Ok, now this looks like an issue with the API key now.
Error when performing Algolia index TypeError [ERR_HTTP_INVALID_HEADER_VALUE]: Invalid value \"undefined\"
Can you check the Algolia logs?
@izanamiah is the issue resolved?
Closing this issue for now.
Had the same issue as the OP (I am on MacOS using the zsh shell) and landed here.
Putting it all on one line and running it in the zsh shell then worked 🥇 :
LOCATION="asia-southeast2" PROJECT_ID=HIDDEN ALGOLIA_APP_ID=HIDDEN ALGOLIA_API_KEY=HIDDEN ALGOLIA_INDEX_NAME=HIDDEN COLLECTION_PATH="FOO/FOO/FOO" FIELDS="id,basics.firstName,basics.lastName,basics.email" GOOGLE_APPLICATION_CREDENTIALS="/Users/ME/APP/credentials.json" npx firestore-algolia-search
@smomin Sorry for my late response. It worked for me eventually. It turns out a problem of my text editor. What happened was when I pasted command from text editor into the terminal, it mess up with some format/char types. Thanks for helping.
Hello, I'm having trouble with importing a firestore collection using the following script:
LOCATION=us-central1\ PROJECT_ID=hidden\ ALGOLIA_APP_ID=hidden\ ALGOLIA_API_KEY=hidden\ ALGOLIA_INDEX_NAME=test\ COLLECTION_PATH=activities\ FIELDS=name;title;description\ GOOGLE_APPLICATION_CREDENTIALS=</path/to/service/account/key>\ npx firestore-algolia-search
Terminal output: {"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"} {"algoliaAPIKey":"****","severity":"INFO","message":"Initializing extension with configuration"}
WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): y {"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read properties of undefined (reading 'split')\n at retrieveDataFromFirestore (/Users/marius/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:76:65)\n at /Users/marius/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:93:9\n at Interface._onLine (node:readline:485:5)\n at Interface._line (node:readline:864:8)\n at Interface._ttyWrite (node:readline:1216:14)\n at ReadStream.onkeypress (node:readline:288:10)\n at ReadStream.emit (node:events:520:28)\n at emitKeys (node:internal/readline/utils:358:14)\n at emitKeys.next ()\n at ReadStream.onData (node:internal/readline/emitKeypressEvents:61:36)"}
npm/npx version is 8.3.1