If Crawler is "Stuck" allow option to ignore current URL in the list and move forward to the next URL

madchops1 / sitemap-generator

Javascript Sitemap Generator

http://botmap.io

32 stars 5 forks source link

If Crawler is "Stuck" allow option to ignore current URL in the list and move forward to the next URL #7

Open lucksp opened 6 years ago

lucksp commented 6 years ago

If the crawler is Stuck on a URL, it would be really slick to allow moving forward on to the next URL in the list. This could be related to current #4 issue

[x] website URL 1
[x] website URL 2
[x] website URL 3 [ X ] website URL 4 (STUCK)
[ ] website URL 5 (next on list)

madchops1 commented 6 years ago

Agreed, shouldn't be too difficult. I gots dis.