search

arcjet / well-known-bots

List of well-known bots and user-agent patterns to detect them
MIT License
3 stars 0 forks source link

Handle dead URLs #8

Open blaine-arcjet opened 3 weeks ago

blaine-arcjet commented 3 weeks ago

Many URLs in the list of bots are dead links. We should decide if we want to remove these or flag them as "probably dead" bots. We could decide to remove them from our list of patterns to reduce Wasm size or we could deny any user agent that contains them since they are suspected to no longer be in service.

davidmytton commented 3 weeks ago

I'd remove them since we can't verify the user agent or categorize them. We could put them under "unknown", but it's probably better to reduce the Wasm size.