search

kezakez / python-web-crawler

39 stars 29 forks source link

skipping? #2

Open demonaxe opened 10 years ago

demonaxe commented 10 years ago

how do you change it to not skip other weblinks?

kezakez commented 10 years ago

it skips processed links and anything that isn't http. https://github.com/kezakez/python-web-crawler/blob/master/crawl.py#L9