Describe the current state/issue
Google Search Console reports that ROR web UI pages are allowed to be crawled, even though the ror search app has a robots.txt that is supposed to be disallowing crawlers.
Describe the desired state/solution
ROR searches and landing pages disallow web crawlers.
Additional notes
Problem may be that redirection of requests only causes requests that look like ROR IDs to go to ror-app. Need to add robots.txt to ror-site and check whether we can regex for ROR ID.
Service/repository ror-app, ror-site
Describe the current state/issue Google Search Console reports that ROR web UI pages are allowed to be crawled, even though the ror search app has a robots.txt that is supposed to be disallowing crawlers.
Describe the desired state/solution
ROR searches and landing pages disallow web crawlers.
Additional notes
Problem may be that redirection of requests only causes requests that look like ROR IDs to go to ror-app. Need to add robots.txt to ror-site and check whether we can regex for ROR ID.