Closed justinormont closed 11 years ago
we need to research what the right way to do this in Node is
Should we just put up a static placeholder in the meantime? Could make a new issue to revamp it at a later date.
On Friday, April 12, 2013, Mohamed Mansour wrote:
we need to research what the right way to do this in Node is
— Reply to this email directly or view it on GitHubhttps://github.com/mohamedmansour/letsgo.io/issues/28#issuecomment-16309413 .
-Joey Hiller 707.456.7374
I think a static file would do well.
The easiest would be creating an endpoint with robots.txt that serves a plain jade file for now. I don't want to expose the entire root as static
echo -e "User-agent: *\nAllow: /" >> public/robots.txt ?
(I can't do it from this computer so if one of you could give it a try? @justinormont maybe?)
oh shoot, you are right, I made public (static)
"http://letsgo.io/: Googlebot can't access your site
Over the last 24 hours, Googlebot encountered 2 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%."