beauchette
commented
1 year ago it's due to chrome 112 with its appropriate chrome driver that fails really fast.
On another hand I still don't understand, with js_render=false, How I can have between 75k and 100k records at times, while the site doesn't really change more than 2/3 pages a day
Description
I have a website with thousands of pages. Most of them are rendered by the server, but some of them use javascript "plugins" and there's no SSR for these features.
So I just installed Typesense 0.24.0 and typesense-docsearch-scraper from git.
My first attempt gets 110k hits in my collection, the second gets me 98k (which is weird, there is not that many changes in an hour) and then anyway, I added "js_render": true to my config, and it only got me 90 hits.
What could cause this ?
Also, there is a lot variation in the number of hits, I tried launching it with _jswait = 2 and got 194 hits, and then another time without even touching config.json and got 90 hits again.
Steps to reproduce
I just set up a new typesense 0.24.0, and the last typesense-docsearch-scraper from git and tried it against my website.
Expected Behavior
I'd expect two consecutive execution to get approximately the same number of hits and I'd expect _jsrender true to cause more hits, because there's more content
Actual Behavior
The discrepancies on the number of hits I described.
Metadata
Typesense: 0.24.0 typesense-docsearch-scraper: latest from git OS: vanilla Ubuntu 22.04