Obscure crawler matching

gams / openkongqi

Outdoor air quality data

Apache License 2.0

3 stars 6 forks source link

Open hrbonz opened 8 years ago

hrbonz commented 8 years ago

A few things that could make the scripts harder to detect:

implement a simili-random scheduling interval to avoid strict patterns in scraping
random/multiple User-Agents