typesense / typesense-docsearch-scraper

A fork of Algolia's awesome DocSearch Scraper, customized to index data in Typesense (an open source alternative to Algolia)
https://typesense.org/docs/guide/docsearch.html
Other
107 stars 39 forks source link

Big variation in number of hits when js_render is true #31

Open beauchette opened 1 year ago

beauchette commented 1 year ago

Description

I have a website with thousands of pages. Most of them are rendered by the server, but some of them use javascript "plugins" and there's no SSR for these features.

So I just installed Typesense 0.24.0 and typesense-docsearch-scraper from git.

My first attempt gets 110k hits in my collection, the second gets me 98k (which is weird, there is not that many changes in an hour) and then anyway, I added "js_render": true to my config, and it only got me 90 hits.

What could cause this ?

Also, there is a lot variation in the number of hits, I tried launching it with _jswait = 2 and got 194 hits, and then another time without even touching config.json and got 90 hits again.

Steps to reproduce

I just set up a new typesense 0.24.0, and the last typesense-docsearch-scraper from git and tried it against my website.

Expected Behavior

I'd expect two consecutive execution to get approximately the same number of hits and I'd expect _jsrender true to cause more hits, because there's more content

Actual Behavior

The discrepancies on the number of hits I described.

Metadata

Typesense: 0.24.0 typesense-docsearch-scraper: latest from git OS: vanilla Ubuntu 22.04

beauchette commented 1 year ago

it's due to chrome 112 with its appropriate chrome driver that fails really fast.

On another hand I still don't understand, with js_render=false, How I can have between 75k and 100k records at times, while the site doesn't really change more than 2/3 pages a day