amazon spider improvements - Githubissues

calgo-lab / green-db

The monorepo that powers the GreenDB.

https://calgo-lab.github.io/green-db/

22 stars 2 forks source link

amazon spider improvements #100

Closed en-GB closed 1 year ago

en-GB commented 1 year ago

adds periodic delays (a few hours every 250 requests)
adds some user agent randomization
strips link tracking data for all pages