search

JayBizzle / Crawler-Detect

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
https://crawlerdetect.io
MIT License
1.99k stars 258 forks source link

Add 9 bots/crawlers #434

Closed Seirdy closed 3 years ago

Seirdy commented 3 years ago

Grepped around my access logs, found 9 new bots/crawlers.

The most useful edition is Pleroma. Pleroma is server software for federated social media, like Mastodon; when users submit links, it generates a link preview using the Open Graph protocol. When other Pleroma instances receive statuses, they too generate link previews. This creates a lot of requests, but it's quite benign and relatively low-volume. I am a user.

Also added Mozilla's HTTP Observatory, which is used to check HTTP headers and TLS configurations for best practices. It's in a category similar to SSL Labs. I am a user.

I'm not too familiar with the other 7, but their user-agents and a quick look-up revealed them to be bots.

Seirdy commented 3 years ago

Pleroma isn't the only social media platform on the "Fediverse"; other platforms also exist and can talk to each other, and many also send requests as they generate link previews. I might add their user-agents later, but this PR as it is rn should be mergeable.

Seirdy commented 3 years ago

force-pushed tabs-to-spaces. ugh.

JayBizzle commented 3 years ago

Looks great, thanks 👍🏻