Make crawling rules configurable via config

let4be / crusty

Broad Web Crawler

GNU General Public License v3.0

83 stars 3 forks source link

Closed let4be closed 3 years ago

let4be commented 3 years ago

Stuff like page limit, max depth, skip no follow links, etc, etc, etc...