search

buckyroberts / Spider

Python website crawler.
https://thenewboston.com/
967 stars 670 forks source link

external links #39

Open mdardir2002 opened 7 years ago

mdardir2002 commented 7 years ago

I'm new to python so excuse my ignorance... You know how is the spider finds all internal links on page and store them in the queue file to be crawled later for more links and so on .... I'm trying to make the spider also save all unique external links in another file called external . Any help!!

ghost commented 7 years ago

The project has been moved to https://github.com/AbdulSheikh/Spider . This project is no longer supported In this github account.