search

monperrus / crawler-user-agents

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:
MIT License
1.19k stars 255 forks source link

Request for new instances #366

Open petrospap opened 2 months ago

petrospap commented 2 months ago

Hello! I have a small request when you add new crawler/s, is it possible to create a separated file with new instances, something like

[
  {
    "info": "Info",
    "created_date": "2024/09/02"
    "totalnewinstances": 1
  },
  {
    "pattern": "Googlebot-Image",
    "newinstances": [
      "Googlebot-Image/2.1"
    ]
  }
]

Reason for this request is, i made a php/javascript script to manage Crawler Agents, and this will help me to update database more easily Script created by your crawler-user-agents.json and follows the same logic, (if you see my page, they exist in category "crawler")

Script also can detect some malicious activity (in javascript)

you can see/download https://simply4all.net/agent-management

Thanks

monperrus commented 1 month ago

this could perhaps be done with an additional CI job, feel free to make a pull-request.