adamdehaven / fetchurls

A bash script to spider a site, follow links, and fetch urls (with built-in filtering) into a generated text file.
https://www.adamdehaven.com/blog/easily-crawl-a-website-and-fetch-all-urls-with-a-shell-script/
MIT License
127 stars 45 forks source link

Fixes #2

Closed vendforce closed 5 years ago

vendforce commented 5 years ago

I suggest you set a trap in your script , so if someone does a ctrl-d or the script fails that the color is reset in the terminal for the user. insert this at the top of your script

resetcolor () { # this is just a rough example is can be implemented better 
echo -e "\033[0m"
}

trap resetcolor INT TERM EXIT

you also need to create the directory that you suggest in the script . it fails to find the directory because its not created,

echo "${COLOR_RESET}# " read -e -p "# Save txt file as: ${COLOR_CYAN}" -i "${filename}" SAVEFILENAME

mkdir -p $savelocation/$SAVEFILENAME # you need to place this here

savefilename=$SAVEFILENAME

I'd suggest you rewrite it , as its a bit messy and wont handle syntax errors properly What if someone has renamed there desktop file or deleted it or wants to put it somewhere else ? crazy I know but your script will fail

what is someone enters an incomplete url or non existent url example : http://www.goodfg.comdwdd you could handle this in a while true loop and check the existence of the url

here is something I threw together as an an example

while true
do
# CHECK IF URL EXISTS AND IS ONLINE
if [[ "$#" -eq  "0" ]] ; then
       echo -e "${YELLOW}Example${NC} : checkurl msn.com"
       echo
       break
       exit
fi

if curl --output /dev/null --silent --head --fail "$CONVERTCASE" then

CHECKURL=$(lynx -dump http://downforeveryoneorjustme.com/$CONVERTCASE | grep -o "It's just you")

if [[ $CHECKURL != 0 ]] ; then
      echo -en "Url exists so lets continue" > /dev/null
      break
else
      echo -en  "Url doesn't exist lets try again" 
      sleep 2
fi
done
adamdehaven commented 5 years ago

@vendforce Thanks for the suggestions! I had some time this morning so I implemented a solution for everything mentioned above. Grab version 2.0 of the code, and give it a try!

vendforce commented 5 years ago

Much better, Nice Work !!!