How to exclude noindex page

c4software / python-sitemap

Mini website crawler to make sitemap from a website.

GNU General Public License v3.0

366 stars 110 forks source link

Open damarkuzz opened 11 months ago

damarkuzz commented 11 months ago

Hi! Could you tell please how to exclude web pages with meta robots "noindex"