adamdehaven / fetchurls

A bash script to spider a site, follow links, and fetch urls (with built-in filtering) into a generated text file.
https://www.adamdehaven.com/blog/easily-crawl-a-website-and-fetch-all-urls-with-a-shell-script/
MIT License
127 stars 45 forks source link

Adding flags, and enabling non-interactive mode #7

Closed adamdehaven closed 4 years ago

adamdehaven commented 4 years ago

This update adds the ability to pass settings (domain, save location, filename, excluded file extension list, etc.) via flags on the command line, which in turn, also enables the script to run in non-interactive mode as suggested by @andrew-acpatt.

This release closes #6 as it includes all of the functionality suggested (and adds some additional features 🎉).