Option to specify bot for robots.txt

c4software / python-sitemap

Mini website crawler to make sitemap from a website.

GNU General Public License v3.0

366 stars 110 forks source link

Closed Garrett-R closed 6 months ago

Garrett-R commented 9 months ago

For a site that has a robots.txt like:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /
Disallow: /login

it would be nice if there was an option to run the script as Googlebot, so it follow those rules.

c4software commented 9 months ago

It's a really nice idea !

I will look at this soon.

Garrett-R commented 9 months ago

Cool! My team can also submit a PR if that helps!