search

yeoman / generator-mobile

Scaffold out a mobile project based on Web Starter Kit
http://yeoman.io
918 stars 99 forks source link

Add `Disallow:` to `robots.txt` #58

Closed arthurvr closed 9 years ago

arthurvr commented 9 years ago

According to the robots.txt standard:

The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below.

Also, Google seems to recommend it.

See the discussion about this in h5bp too

kevva commented 9 years ago

Could you update the URL too? https://github.com/h5bp/html5-boilerplate/commit/db52584e8c9e1beb055681ea0e1580fcca84e259#diff-aa2646a667ee1cd83235786dccef4a26R1.

arthurvr commented 9 years ago

@kevva Done!

kevva commented 9 years ago

Hehe, could you squash them together too :)? Just add a message to the first commit about the URL update.

arthurvr commented 9 years ago

@kevva Jup. Done.

kevva commented 9 years ago

Thanks :)!