Closed arthurvr closed 9 years ago
Same thing as https://github.com/yeoman/generator-angular/commit/fd942ce9a29b1f1fef645f61e6932140957d4d81.
According to the robots.txt standard:
The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below. "
This is also encouraged by the major search engines.
Ref h5bp/html5-boilerplate#1487 http://www.robotstxt.org/orig.html
Same thing as https://github.com/yeoman/generator-angular/commit/fd942ce9a29b1f1fef645f61e6932140957d4d81.
According to the robots.txt standard:
This is also encouraged by the major search engines.
Ref h5bp/html5-boilerplate#1487 http://www.robotstxt.org/orig.html