search

rivermont / spidy

The simple, easy to use command line web crawler.
GNU General Public License v3.0
334 stars 69 forks source link

Always download from wikipedia #78

Closed Jerry4013 closed 3 years ago

Jerry4013 commented 3 years ago
rivermont commented 3 years ago

All of the default config files (https://github.com/rivermont/spidy/tree/master/spidy/config) start on Wikipedia. You can change the default starting link(s) by editing one of the config files, or creating your own. Wikipedia is a somewhat arbitrary starting location but unless there's a good reason to use a different link it will stay.