spatie / crawler

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.
https://freek.dev/308-building-a-crawler-in-php
MIT License
2.51k stars 357 forks source link

setConcurrency not working if using browsershoot #399

Closed lenusch closed 2 years ago

lenusch commented 2 years ago

Hi, i can't explain so good, but is it possible to speed up the crawl? We wait hours to finish big Sites. We need browsershoot to run, because we want to crawl .js files. Can't we set something which is running concurrently instead of one by one. Good tool but in the End really slow, with our Usecase. Problem is we need browsershoot as JS is often loaded a little bit later.

BTW: you can set any number you want with setConcurrency if will only start the browser once. I wont start several Instance. We also see no corresponding code for that.