Closed junecv closed 5 months ago
When you changed the extension configuration, could you confirm that Firebase re-deployed both the backfill function and the indexingOnWrite function?
Could you try re-installing the extension from scratch with the new field added, and see if that helps?
Also, do you see any logs in the backfill functions in the Firebase web console, showing any errors?
Apologies I said there was no attempt by the extension function to update. I couldn't find anything under indexToTypesenseOnFirestoreWrite function log, but now I found below under backfillToTypesenseFromFirestore function log. Let me know if you find something below, and still want me to try re-installing. I probably will try with a new, smaller setup as I don't want to disrupt the live app.
--- backfilling function triggered ---
{
"textPayload": "Backfilling [a, b, c (last one newly added)] fields in Firestore documents from products into Typesense Collection products on xyz.typesense.net",
"insertId": "insertID",
"resource": {
"type": "cloud_function",
"labels": {
"region": "us-central1",
"project_id": "projectID",
"function_name": "ext-firestore-typesense-search-backfillToTypesenseFromFirestore"
}
},
"timestamp": "2024-01-10T04:09:17.988118Z",
"severity": "INFO",
"labels": {
"execution_id": "executionID",
"runtime_version": "nodejs16_20231231_16_20_2_RC00",
"instance_id": "instanceID"
},
"logName": "projects/projectID/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
"trace": "projects/projectID/traces/traceID",
"receiveTimestamp": "2024-01-10T04:09:18.307724632Z"
}
Following that is an error:
Import error ImportError: 0 documents imported successfully, 1000 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
at ImportError.TypesenseError [as constructor] (/workspace/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
at new ImportError (/workspace/node_modules/typesense/lib/Typesense/Errors/ImportError.js:25:28)
at Documents.<anonymous> (/workspace/node_modules/typesense/lib/Typesense/Documents.js:154:39)
at step (/workspace/node_modules/typesense/lib/Typesense/Documents.js:48:23)
at Object.next (/workspace/node_modules/typesense/lib/Typesense/Documents.js:29:53)
at fulfilled (/workspace/node_modules/typesense/lib/Typesense/Documents.js:20:58)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5) {
importResults: [
{
code: 409,
document: '{"document content"}',
error: 'A document with id <DocID> already exists.',
success: false
},
....
Then these two messages:
Done backfilling to Typesense from Firestore
Function execution took 9275 ms, finished with status: 'ok'
And this error again:
Import error ImportError: 0 documents imported successfully, 7366 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
at ImportError.TypesenseError [as constructor] (/workspace/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
at new ImportError (/workspace/node_modules/typesense/lib/Typesense/Errors/ImportError.js:25:28)
at Documents.<anonymous> (/workspace/node_modules/typesense/lib/Typesense/Documents.js:154:39)
at step (/workspace/node_modules/typesense/lib/Typesense/Documents.js:48:23)
at Object.next (/workspace/node_modules/typesense/lib/Typesense/Documents.js:29:53)
at fulfilled (/workspace/node_modules/typesense/lib/Typesense/Documents.js:20:58)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5) {
importResults: [
{
code: 409,
document: '{"document content"}',
error: 'A document with id <Doc ID> already exists.',
success: false
},
...
Ah this is because during a backfill, we do action=create
, instead of action=upsert
, so Typesense throws an error saying the document being backfilled already exists.
We didn't account for this use-case of backfilling a 2nd time when a new field is added to an existing collection. Will push out an update for this shortly.
Yay! Thanks for arranging update for this. That would be very useful. A kind note, please also take into account the user-case that a field is removed (if that's not too much to ask).
Could you try installing this pre-release version of the extension and let me know how it goes:
A kind note, please also take into account the user-case that a field is removed (if that's not too much to ask).
Yup, action=upsert
does this automatically
I did, below steps and result.
Edit: no logs available
- upon completion, nothing show up in typesense, even though I had a typesense_sync backfill collection in place (before this whole setup)
This is expected, because only after the extension is installed, it starts listening for the creation of the typesense_sync backfill collection. It does not look at an existing typesense_sync collection.
- upon completion, typesense is not updated, all docs still have 2 fields only
This is also expected, since the backfill only runs when explicitly triggered by creating the typesense backfill collection. It does not run automatically when the extension is re-configured / installed / re-installed.
So it sounds like things are working as designed, since all docs in Typesense now have 3 fields in Step 10?
Yes. I also tried to remove one field in config, and all work out as expected. Thanks for your lightning-fast communication and implementation!
Happy to help! And thank you for all the feedback.
I'll publish this version live shortly.
Description
Added field in under Extension configuration did not trigger document updates in Typesense
Steps to reproduce
Under Extension Configuration in Firebase extension, add a field to the existing list of fields under "Firestore Collection Fields (Optional)".
Expected Behavior
Expect the /typesense_sync/backfill/trigger:true collection takes care of the change.
Actual Behavior
There was no attempt by the extension function to update any document in Typesense. Tried to remove and recreate the backfill collection, still nothing. The change is only reflected when a document is actually updated in firebase.
Metadata
Typesense Version: <1.2.0
OS: MacOS 14.2.1 (23C71)