search

SpiderLabs / owasp-modsecurity-crs

OWASP ModSecurity Core Rule Set (CRS) Project (Official Repository)
https://modsecurity.org/crs
Apache License 2.0
2.45k stars 726 forks source link

Added SemrushBot to crawlers-user-agents.data #1672

Closed dune73 closed 4 years ago

fzipi commented 4 years ago

LGTM.

fzipi commented 4 years ago

Just to complement:

To stop SEMrushBot from crawling your site, add the following rules to your robots.txt file:

    To block SEMrushBot from crawling your site for a webgraph of links:

        User-agent: SemrushBot
        Disallow: /