add a script to generate robots.txt

monperrus / crawler-user-agents

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:

MIT License

1.13k stars 243 forks source link

Open monperrus opened 4 years ago

monperrus commented 4 years ago

It would be great to be able to generate a valid robots.txt from the JSON file, see https://github.com/monperrus/crawler-user-agents/issues/169#issuecomment-620149874