Deleting a field in Firestore is sending a partialUpdateObject request with the field included

[dallasclark]

Related Algolia Support Request: 488433 Support Agent: Tatsuro Handa

We're seeing issues when deleting fields in a Firestore document and Algolia is still showing the field on the object. When deleting the field, we're expecting the Algolia object to delete the attribute as well.

Through Algolia Support, we have identified the attribute is deleted through a updateObject request, but the firestore-algolia-search extension is immediately sending a partialUpdateObjectrequest with the field included.

We have a fairly vanilla configuration, the Firestore Algolia Search Extension has been installed to sync a collection from Firestore with no specified indexable fields (left blank). We have also experimented with indexable fields and received the same result. There is no additional code for indexing with Algolia as the extension is handling the sync.

Algolia Support Team has read access to our account until Tuesday, May 10, 2022.

[smomin]

hey @dallasclark

I have taken a look at your issue but I was not able to replicate it. Here is my video of what I did. Let me know if I need to adjust me steps to replicate the issue.

https://user-images.githubusercontent.com/1305323/165221071-a6a71247-bb91-400f-91e4-78dd6ea24ec5.mov

[dallasclark]

Hi @smomin,

Thanks for investigating. Unfortunately, it doesn't appear to be easily replicated but we're definitely seeing issues when we start to get into larger data sets (i.e. production).

Are there any limitations to the number of documents that can be updated at once? Can race conditions occur? We're using batch commits to the collections to re-rank the songs and apply new rankings. If the ranking is above 100, the ranking field is deleted on the Firestore document.

This an example of many of the end result:

Firestore Document (recordingid: 17e5cfb9-9107-481f-a8aa-33fc8e9b410e)

Note the absence of the ranking field.

Algolia Object (ID: 17e5cfb9-9107-481f-a8aa-33fc8e9b410e)

Our songs collection in Firestore is appearing correctly, as shown below with ranking in descending order:

Yet, Algolia has a number of objects in the index that are maintaining the ranking field that were deleted on the Firestore document.

[smomin]

hey @dallasclark

just to step back for a second. how are you running the process to update the Algolia index? also, can you share you logs with me. thanks.

sajid

[dallasclark]

Hey @smomin,

We're using the https://github.com/algolia/firestore-algolia-search extension, as shown above, it's a very vanilla setup. We're relying on this extension only to sync data between Firestore and Algolia.

Logs coming.

DC

[dallasclark]

No errors showing ...

Example output

11:39:17.626 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:17.627 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1810 ms, finished with status: 'ok'
11:39:17.628 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:17.628 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:17.631 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document 200f315d-f13d-4c4e-ba3c-901c785c96a7
11:39:17.632 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute partialUpdateObject' ]
11:39:17.670 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:17.670 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:17.671 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:17.672 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:17.672 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:17.673 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:17.673 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:17.673 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:17.673 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:17.677 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document 397dbf34-78d2-46cd-8c3e-042718e314f9
11:39:17.677 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document 49c63c74-50b6-4545-aa90-1cd52cf1d152
11:39:17.677 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute saveObject' ]
11:39:17.677 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute partialUpdateObject' ]
11:39:17.681 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document 40972314-1df8-4b40-8956-d06dc478a2c4
11:39:17.681 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute partialUpdateObject' ]
11:39:17.700 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 4572 ms. Finished with status: ok
11:39:17.731 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 4277 ms. Finished with status: ok
11:39:17.732 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1235 ms, finished with status: 'ok'
11:39:17.780 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1217 ms, finished with status: 'ok'
11:39:17.835 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 4590 ms. Finished with status: ok
11:39:17.867 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 4633 ms. Finished with status: ok
11:39:17.881 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 4532 ms. Finished with status: ok
11:39:18.014 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:18.029 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:18.049 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:18.103 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1316 ms. Finished with status: ok
11:39:18.122 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:18.124 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:18.124 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:18.126 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1288 ms. Finished with status: ok
11:39:18.129 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document 38fb83ca-cb3f-4feb-b967-cd7a5f5c29c3
11:39:18.129 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute partialUpdateObject' ]
11:39:18.308 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1245 ms, finished with status: 'ok'
11:39:18.598 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:18.598 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:18.640 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1513 ms. Finished with status: ok
11:39:18.656 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution started
11:39:18.668 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:18.670 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:18.670 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:18.673 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document ce03f9ef-ba85-4790-a776-7b6a2bcca792
11:39:18.673 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute saveObject' ]
11:39:18.751 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:18.766 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1316 ms. Finished with status: ok
11:39:18.778 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution started
11:39:18.820 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1733 ms. Finished with status: ok
11:39:18.871 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:18.874 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:18.874 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:18.877 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document d766acba-1649-4723-8249-e652d7d3d3cf
11:39:18.877 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute partialUpdateObject' ]
11:39:18.970 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:18.972 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:18.972 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:18.976 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document 2ff3797f-28a1-4557-973b-e069e4e0458f
11:39:18.976 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute partialUpdateObject' ]
11:39:19.255 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 476 ms. Finished with status: ok
11:39:19.272 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 615 ms. Finished with status: ok
11:39:23.626 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution started
11:39:25.216 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:25.275 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:25.278 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:25.279 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:25.281 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document f943e9c9-05dd-49b1-ac9a-33e1839c359d
11:39:25.281 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute saveObject' ]
11:39:25.435 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1809 ms. Finished with status: ok
11:39:37.082 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution started
11:39:37.112 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution started
11:39:37.123 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution started
11:39:38.661 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:38.663 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:38.670 AM
ext-firestore-algolia-search-executeIndexOperation
Initializing extension with configuration
11:39:38.698 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1586 ms. Finished with status: ok
11:39:38.702 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1579 ms. Finished with status: ok
11:39:38.709 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution started
11:39:38.709 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution started
11:39:38.825 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:38.828 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:38.828 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:38.831 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document 0abb5d2e-e109-44d7-a8d0-77f881594cf1
11:39:38.831 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute saveObject' ]
11:39:38.841 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:38.841 AM
ext-firestore-algolia-search-executeIndexOperation
Started extension execution with configuration
11:39:38.844 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:38.844 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'fields: ' ]
11:39:38.844 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:38.845 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'Detected a change, execute indexing' ]
11:39:38.848 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document c92e7dac-d642-4f02-9dc3-05700201370c
11:39:38.848 AM
ext-firestore-algolia-search-executeIndexOperation
Updating existing Algolia index for document e66198a5-6735-43dd-807f-7d749f02a8f7
11:39:38.848 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute saveObject' ]
11:39:38.848 AM
ext-firestore-algolia-search-executeIndexOperation
[ 'execute saveObject' ]
11:39:39.036 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 1953 ms. Finished with status: ok
11:39:39.183 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 473 ms. Finished with status: ok
11:39:39.198 AM
ext-firestore-algolia-search-executeIndexOperation
Function execution took 488 ms. Finished with status: ok

DC

[smomin]

hey @dallasclark

would a batch is sent to firestore, would a batch have the same document updated multiple times? I assume yes, but I want to make sure.

[dallasclark]

Hi @smomin,

Our first batch script updates all documents once with a score, and removes every single ranking.

We then have a timeout for 5 seconds before the first 100 documents are fetched (ordered by score), these documents are then updated in the next batch script with rankings 1 to 100.

We’re using Google Cloud Scheduled functions to execute these batch commits every 10 minutes.

Sometimes it works, sometimes it doesn’t. Although Firestore is correct every single time.

Occasionally we see gaps on Algolia in the ranking like 1, 2, 3, 5. Other times the ranking is removed in Firestore but not Algolia, and requires manual intervention to fix.

We’ve also tried removing batch scripts and updating one document at a time, including 200 - 1000 ms timeouts between each document update with no luck. Eventually, Algolia is out of sync.

You can see the results from Algolia at https://splashhq.com/charts. At the time of writing this message, I’ve checked Firestore and it’s correct but Algolia is out of sync.

DC

[smomin]

@dallasclark I am going to introduce a minor feature that will allow a force data sync before processing the data to Algolia. The force data sync would be another hit to the Firestore db so I am going to make this a configuration toggle so it can be turned on and off. We can try this feature to see if it correct the data sync issue between Firestore/Algolia. I will let you know when its in place.

[dallasclark]

Hey @smomin,

Looking at the Pull Request (https://github.com/algolia/firestore-algolia-search/pull/108), it appears you're identifying the issue from having 2 consecutive calls to the document and causing a race condition of some sort.

How confident are you if we updated over 100,000 unique documents in Firestore in a single batch commit, only once per document?

DC

[dallasclark]

Hi @smomin,

Early indicators are showing that a single batch commit of all documents is working properly compared to separating the logic into 2 separate batch commits. Unfortunately, memory usage increases significantly with this approach so it's not a long-term fix but it does resolve the issue temporarily.

DC

[smomin]

Hey @dallasclark

I have pushed an update to the extension. In the configuration, there is a yes/no toggle to force a Firestore read before sending data to Algolia. Would you be interested in trying this new update? Let me know.

Sajid

[smomin]

I am closing this issue. I have a configuration to allow for a force data update before being indexed to Algolia.