search

mohamedmansour / hackathon-letsgo.io

letsgo.io - Photo Hack Day 4 project
http://letsgo.io
5 stars 6 forks source link

Create robots.txt #28

Closed justinormont closed 11 years ago

justinormont commented 11 years ago

"http://letsgo.io/: Googlebot can't access your site

Over the last 24 hours, Googlebot encountered 2 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%."

mohamedmansour commented 11 years ago

we need to research what the right way to do this in Node is

jthiller commented 11 years ago

Should we just put up a static placeholder in the meantime? Could make a new issue to revamp it at a later date.

On Friday, April 12, 2013, Mohamed Mansour wrote:

we need to research what the right way to do this in Node is

— Reply to this email directly or view it on GitHubhttps://github.com/mohamedmansour/letsgo.io/issues/28#issuecomment-16309413 .

-Joey Hiller 707.456.7374

justinormont commented 11 years ago

I think a static file would do well.

mohamedmansour commented 11 years ago

The easiest would be creating an endpoint with robots.txt that serves a plain jade file for now. I don't want to expose the entire root as static

jeremybenaim commented 11 years ago

echo -e "User-agent: *\nAllow: /" >> public/robots.txt ?

jeremybenaim commented 11 years ago

(I can't do it from this computer so if one of you could give it a try? @justinormont maybe?)

mohamedmansour commented 11 years ago

oh shoot, you are right, I made public (static)