search

madchops1 / sitemap-generator

Javascript Sitemap Generator
http://botmap.io
32 stars 5 forks source link

Prevent pdf, images, etc... #4

Open madchops1 opened 6 years ago

lucksp commented 6 years ago

Is there a way to ignore PDFs at this point? It does get stuck

EDIT: add Disallow: /pdf into the UI where robots.txt would be

lucksp commented 6 years ago

Also, SVGs create an error, but they do get skipped:

Error: <polygon> attribute points: Expected number, "\\\"313.11".
a.getLinks @ scripts.24d0890a.js:1