search

typesense / firestore-typesense-search

Firebase Extension to automatically push Firestore documents to Typesense for full-text search with typo tolerance, faceting, and more
https://extensions.dev/extensions/typesense/firestore-typesense-search
Apache License 2.0
159 stars 35 forks source link

Backfill is crashing #82

Closed rohan-mehta closed 2 months ago

rohan-mehta commented 4 months ago

Description

I have a large-ish firebase collection (hundreds of thousands of docs) that I'm trying to sync to typesense. When I start the backfill, I see:

Memory limit of 2048 MiB exceeded with 2054 MiB used. Consider increasing the memory limit, see https://cloud.google.com/functions/docs/configuring/memory

How do I fix this?

Metadata

Typesense Version: 26

OS: typesense cloud

rohan-mehta commented 4 months ago

Based on some closed issues — seems like the extension tries to load the entire collection into memory instead of paginating?

jasonbosco commented 2 months ago

I just merged #70, which contains a fix for this issue by backfilling in batches.

This is now available in v1.5.0-rc.1. The installation link for this RC build is in the link above.

Could you give it a shot and let me know if the issue still persists?