Update robots.txt file to stop legitimate services from unnecessary scanning paths such as the api

[WadeBarnes]

For example we're seeing the API being scanned by Bytespider.

The robots.txt file is defined, but does not specify any rules.

Are there other setting/files we can use to deter legitimate services from scanning the API unnecessarily?

[WadeBarnes]

In this case the organization behind Bytespider is well known for not respecting the robots.txt file, but we need to update it anyway.

[WadeBarnes]

From @cvarjao: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#syntax

[cvarjao]

The Disallow path should start with /

User-agent: *
Disallow: /

Some other comments: https://www.feitsui.com/en/article/32

It seems Bytespider and Sogou spiders are not fully compatible with robots exclusion standard. These crawlers magically disappeared one week after I created a separate block for each user agent in robots.txt.

[WadeBarnes]

Other sources indicate they do not respect the robots.txt file at all.

• https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/
• https://www.reddit.com/r/flask/comments/161fqml/what_to_do_about_bytespider/
• https://stackoverflow.com/questions/57908900/what-is-the-bytespider-user-agent

[WadeBarnes]

That said it's worth adding the explicit entries into the robots.txt file to try things out.