search

typesense / typesense-docsearch-scraper

A fork of Algolia's awesome DocSearch Scraper, customized to index data in Typesense (an open source alternative to Algolia)
https://typesense.org/docs/guide/docsearch.html
Other
101 stars 36 forks source link

Possible to send a header with `docker run -i ... typesense/docsearch-scraper` ? #45

Open paulrudy opened 1 year ago

paulrudy commented 1 year ago

I'm running typesense on fly.io, and using a github action to run typesense-docsearch-scraper via docker run -i ... typesense/docsearch-scraper.

My typesense app runs on a couple of VM's, which are not an HA cluster, but operate as single-node clusters. I'd like to be able to target a specific VM when I run the scraper. On fly, the way to do that is to send a header with the request: Fly-Prefer-Region: <region-code>

Is there a method to include such a header when running the scraper?

jasonbosco commented 1 year ago

The scraper doesn't have a way to pass in custom headers. But happy to accept a PR that adds support for this.

paulrudy commented 1 year ago

Thanks. It's far above my ability to code a PR for that functionality, but perhaps headers can be passed in as parameters in a query string added to the start_urls and end_urls in the scraper config?