typesense / typesense-docsearch-scraper

A fork of Algolia's awesome DocSearch Scraper, customized to index data in Typesense (an open source alternative to Algolia)
https://typesense.org/docs/guide/docsearch.html
Other
100 stars 36 forks source link

feat: carry curation rules and synonyms to new collection after scraper runs #66

Closed tharropoulos closed 3 weeks ago

tharropoulos commented 2 months ago

Change Summary

Updates the scraper workflow to ensure that user-defined synonyms and curation rules are preserved across runs. Specifically, it adds functionality to copy these rules from the old collection to the new collection before the old collection is deleted. This change helps maintain user configurations without disruption during the scraping process.

Changes include:

PR Checklist

tharropoulos commented 3 weeks ago

Just saw that pylint now fails the check because of duplicate code on the test files. It has to do with not syncing the lock file with the Pipfile and having an older version of pylint as a result. Should we break this out on fixtures to avoid rewrites, or add a rule to ignore the duplicate code message in the pylint config file?

jasonbosco commented 3 weeks ago

This is now available in v0.10.0