search

crystal-community / crystal-libraries-needed

A list of libraries that are needed or wanted for the Crystal-Language
142 stars 4 forks source link

A port of, or comparable functionality too Ruby Robotstxt #57

Open johnjansen opened 7 years ago

johnjansen commented 7 years ago

see https://github.com/gjtorikian/robotstxt-parser

johnjansen commented 7 years ago

see

https://yoast.com/ultimate-guide-robots-txt/ https://en.wikipedia.org/wiki/Robots_exclusion_standard

separately consider sitemaps https://en.wikipedia.org/wiki/Sitemaps