typesense / firestore-typesense-search

Firebase Extension to automatically push Firestore documents to Typesense for full-text search with typo tolerance, faceting, and more
https://extensions.dev/extensions/typesense/firestore-typesense-search
Apache License 2.0
159 stars 35 forks source link

Support for subcategory backfill (redo) #93

Open vazome opened 1 month ago

vazome commented 1 month ago

Change Summary

@vazome – Improves upon https://github.com/typesense/firestore-typesense-search/pull/72. I took into account that orginal file was renamed and put subcategory backfill logic explanation into REAME. Tested this change in Cloud Functions, it works.

@guenth39 – With these changes, the backfill will now work also for subcollections, not only for "root" collection. If the collection path is something like users/{userId}/items the sync has already worked, but the backfill did not as the documents were queried with this collection path, but that did not work. Instead, we use the collectionGroup feature now with only the last part of the path.

PR Checklist

vazome commented 1 month ago

It closes: https://github.com/typesense/firestore-typesense-search/issues/17

vazome commented 1 month ago

FYI @guenth39 @jasonbosco

I have fixed previous issues

vazome commented 4 weeks ago

@jasonbosco could you please elaborate, I should make a test in the code itself or something different?

UPD: okay let's see

jasonbosco commented 4 weeks ago

You want to create a new test file like this one that reads the collection name configuration from new test params file like this, but has a nested collection name specified in it.

And then update the test runner here and add a test part 3 may be, to run the test in CI

vazome commented 2 weeks ago

@jasonbosco tests included, please verify

UPD: fixed syntax error

vazome commented 2 weeks ago

Okay, I see the errors, forgot to do ESlint

vazome commented 2 weeks ago

Corrected the wrong indent and missing new line

tharropoulos commented 2 weeks ago

Hey, I saw that the tests were failing, so I took a closer look and tried to debug it myself. Turns out, if you log the value:

      const typesenseDocsStr = await typesense.collections(encodeURIComponent(typesenseSubcollectionName)).retrieve();
      console.dir(typesenseDocsStr);

The collection has no documents present:

    {
      created_at: 1731922232,
      default_sorting_field: '',
      enable_nested_fields: true,
      fields: [
        {
          facet: false,
          index: true,
          infix: false,
          locale: '',
          name: '.*',
          optional: true,
          sort: false,
          stem: false,
          store: true,
          type: 'auto'
        }
      ],
      name: 'books_firestore/1/subcollection',
      num_documents: 0,
      symbols_to_index: [],
      token_separators: []
    }

What's also interesting is one in two runs, the test fails to delete the existing collection, resulting in a 409 from the server