search

adamdehaven / adamdehaven-com-comments

Comments for adamdehaven.com (powered by utterances)
https://www.adamdehaven.com
0 stars 0 forks source link

Easily crawl a website and fetch all urls from the command line #5

Open utterances-bot opened 9 months ago

utterances-bot commented 9 months ago

Easily crawl a website and fetch all urls from the command line | AdamDeHaven.com

A bash script to spider a site, follow links, and fetch urls (with built-in filtering) into a generated text file.

https://www.adamdehaven.com/blog/easily-crawl-a-website-and-fetch-all-urls-with-a-shell-script/

stevebab commented 9 months ago

Hi Adam. Thank you for posting this. Worked really well. I have a use case where I'm trying to get the outbound links as well. Do you know how I might change your script to also get the urls to external links on the site?