search

TeamHG-Memex / autopager

Detect and classify pagination links
99 stars 25 forks source link

detect pagination options links #2

Open kmike opened 8 years ago

kmike commented 8 years ago

Websites often provide links "show 20/50/100 results per page"; by following them crawler can get the same contents multiple times. It'd be nice to detect these links.