search

wummel / linkchecker

check links in web documents or full websites
http://wummel.github.io/linkchecker/
GNU General Public License v2.0
1.42k stars 234 forks source link

Fix cache: Don't check one url multiple times #687

Open PetrDlouhy opened 7 years ago

PetrDlouhy commented 7 years ago

As mentioned in #608, Linkchecker does multiple checks for one URL. It is caused by inconsistent use of cache. The URL is added to the queue when checking page, but it doesn't get to cache until the moment of checking the URL itself. In the meantime, it can be added to queue multiple times.

This PR should fix it.

PetrDlouhy commented 7 years ago

Travis tests are working, so only tests are waiting.

PetrDlouhy commented 7 years ago

Now it should be finished. It includes tests for urlqueue.

anarcat commented 7 years ago

please reroll this PR in the new organisation here: https://github.com/linkcheck/linkchecker/pulls see #686 for details

dpalic commented 7 years ago

would be a great feature to merge into mainstream. Is there any timeplan for this? The current checking is too much load generating to the target page/server for checking over and over the same URL