Scrapy spider 3x slower after implementing this

[petrrutz]

Hi, I apologize, I know this is probably the wrong place to post this, but I have the following issue: After implementing this in my spider, my average number of crawled pages per minute is now about three times slower. I have CONCURRENT_REQUESTS_PER_DOMAIN set to 32. I am trying to find out what the bottleneck is and what I can possibly do to regain the previous speed (approximately three times faster) without overloading the site.

I appreciate your work on this project and any advice you can give me.

[jxlil]

Hi @petrrutz

Could you share a small spider that works as a PoC? I could review this later.

Thanks

[jxlil]

You're right, I was doing some tests with 100 random websites and a CONCURRENT_REQUESTS of 5 and scrapy-impersonate is much slower.

Twisted

scrapy crawl testing  3.43s user 1.03s system 25% cpu 17.223 total
scrapy crawl testing  3.67s user 1.13s system 27% cpu 17.578 total

scrapy-impersonate

scrapy crawl testing  3.44s user 1.10s system 19% cpu 23.003 total
scrapy crawl testing  3.57s user 1.18s system 19% cpu 24.631 total

[petrrutz]

I'm sorry I did not get back to you sooner. My spider is complex, and I was pushed to make it work again, so I needed to prioritize. I appreciate you looking into it. In the end, I used a similar library: https://github.com/tieyongjie/scrapy-fingerprint. It is also slower than native Scrapy, but faster than scrapy-impersonate, so maybe you can compare both.

[jxlil]

I just generated a PR to fix this, now the results are similar to Twisted. Testing was performed with a CONCURRENT_REQUESTS of 32 and 84 random sites

Twisted

test1: scrapy crawl impersonate  2.62s user 0.55s system 33% cpu 9.344 total
test2: scrapy crawl impersonate  2.29s user 0.39s system 46% cpu 5.814 total

scrapy-impersonate

test1: scrapy crawl impersonate  2.29s user 0.47s system 35% cpu 7.854 total
test2: scrapy crawl impersonate  2.25s user 0.45s system 49% cpu 5.444 total