Closed reezer closed 11 years ago
What do you mean?
Like that:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=183669
When the sitemap has been fetched it could for example provide an array of sitemaps.
Is that what you mean?
var robots = require('robots')
, parser = new robots.RobotsParser();
parser.setUrl('http://nodeguide.ru/robots.txt', function(parser, success) {
if(success) {
parser.getSitemaps(function (sitemaps) {
/*
sitemaps is an array of links like:
http://nodeguide.ru/sitemap_1.xml
http://nodeguide.ru/sitemap_2.xml
*/
sitemaps.forEach(function(item){
console.log(item);
});
});
}
});
Right. Just think it would be amazing if there was a library that did parse all the robots.txt stuff in one go.
Yes, it's a good idea. I'll try to add this feature in the next couple of weeks.
What about also providing a function returning the links to Sitemaps?