search

jxlil / scrapy-impersonate

Scrapy download handler that can impersonate browser' TLS signatures or JA3 fingerprints.
MIT License
78 stars 9 forks source link

Fix: Scrapy spider 3x slower after implementing this #11

Closed jxlil closed 1 month ago

jxlil commented 1 month ago

scrapy-impersonate is now faster. The following tests were performed with a CONCURRENT_REQUESTS of 32 and 84 random websites:

Before:

scrapy crawl impersonate  3.57s user 0.64s system 29% cpu 14.467 total

After:

scrapy crawl impersonate  2.32s user 0.48s system 52% cpu 5.323 total

Closes https://github.com/jxlil/scrapy-impersonate/issues/10