Closed shriphani closed 10 years ago
Crawler must obey robots.txt. User can still override this setting.
Merged, thanks!
Crawler must obey robots.txt. User can still override this setting.