Add AI agents - Githubissues

monperrus / crawler-user-agents

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:

MIT License

1.19k stars 255 forks source link

Open httpjamesm opened 10 months ago

httpjamesm commented 10 months ago

AI tools like ChatGPT can crawl webpages. Their user agents should be added to this list.

https://darkvisitors.com is creating an index of AI crawler user agents.

monperrus commented 10 months ago

yes, great idea!