I'm using this package right (more than 500k pages) now and it would be wonderful to have those features:
1) Write sitemap while crawling. That way user knows everithing is working. At the moment I can only see it's crawling looking at the webserver log. But I have no Idea if, when it ends, it will really write the sitemap or not. And it can took hours (or days).
2) Use a log with already fetched urls (or the sitemap itself) with an option to restart without doing it from the beginning. But don't know if this is possible due the need to crawl anyway to be sure to include new pages. But if the user knows there are not new pages and he just needed to interrupt the crawler he could restart from (let say) the last 100 crawlder urls.
I'm using this package right (more than 500k pages) now and it would be wonderful to have those features: 1) Write sitemap while crawling. That way user knows everithing is working. At the moment I can only see it's crawling looking at the webserver log. But I have no Idea if, when it ends, it will really write the sitemap or not. And it can took hours (or days). 2) Use a log with already fetched urls (or the sitemap itself) with an option to restart without doing it from the beginning. But don't know if this is possible due the need to crawl anyway to be sure to include new pages. But if the user knows there are not new pages and he just needed to interrupt the crawler he could restart from (let say) the last 100 crawlder urls.