search

hmol / LinkCrawler

Find broken links in webpage
MIT License
117 stars 59 forks source link

Moved URL's aren't crawled #24

Open n3wjack opened 8 years ago

n3wjack commented 8 years ago

A url that returns a 301 (moved permanently) doesn't get crawled afterwards. I seem to be having a bunch of those where url's not ending in a slash give a 301 to the page containing the trailing slash.

I'm guessing this is not intended. If not, let me know and I have some code ready that fixes this which needs some cleanup and perhaps some unit tests.

leglock commented 7 years ago

This is also happening with 302 responses.

tdwright commented 7 years ago

Hi @n3wjack.

Sorry about the delay in getting back to you - didn't spot this issue.

If you've got code that's "nearly there", perhaps we could start a feature branch and work on it there?

Let me know and I'll create a branch for you.

Tom