The web application contains a robots.txt file. This file is used to give instructions to web robots (also called crawlers or spiders) to ignore specified files or directories when indexing a site. This can be useful for keeping certain areas of your site out of search engine results and can be beneficial for search engine optimisation (SEO). While well-behaved robots (such as the search engine robot used by Google to index websites) will abide by rules provided in robots.txt, robots can choose to ignore this file, making it unsuitable as an access control measure. More importantly, all information in robots.txt is publicly accessible, meaning that visitors to your site can tell which directories you have specified that robots should ignore. The robots.txt file should therefore not contain any information about sensitive files in the installation, and should not be relied upon to prevent indexing of the web application by badly-behaved web robots.
The web application contains a robots.txt file. This file is used to give instructions to web robots (also called crawlers or spiders) to ignore specified files or directories when indexing a site. This can be useful for keeping certain areas of your site out of search engine results and can be beneficial for search engine optimisation (SEO). While well-behaved robots (such as the search engine robot used by Google to index websites) will abide by rules provided in robots.txt, robots can choose to ignore this file, making it unsuitable as an access control measure. More importantly, all information in robots.txt is publicly accessible, meaning that visitors to your site can tell which directories you have specified that robots should ignore. The robots.txt file should therefore not contain any information about sensitive files in the installation, and should not be relied upon to prevent indexing of the web application by badly-behaved web robots.
Severity: Informational
CVSS Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:N
CVSS Score: 0.0
Recommendation: Audit the robots.txt file for inclusion of any sensitive information.