search

KnowledgeLinks / alliance-bibcat

Colorado Alliance of Research Libraries BIBCAT Project
3 stars 0 forks source link

Google and robots.txt file #15

Closed swalker76 closed 6 years ago

swalker76 commented 6 years ago

Our Google Search Console reports the following on August 22:

robots.txt fetch failed You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file.

I can open it in a web browser - looks like this "User-agent: * Disallow:"

jermnelson commented 6 years ago

Found this stackoverflow response that the Content-Type for robots.txt should be text/plain. In commit, added new route that returns robots.txt with the text/plain content-type.

swalker76 commented 6 years ago

Issue is now resolved.