Open Simbiat opened 10 months ago
Hm, that one did not cover the questions above, in the end, although it did mention multiple feed bots, and it resulted in code for validating categories. I am, essentially, talking about cleaning up the types.
I guess we don't have a "clean" definition of categories to use. Feel free to create a PR to clean them up a bit.
I can add this to #7490. Or would a separate PR be better?
@Simbiat It's better to have a separate PR, as that makes reviewing easier.
I've come across https://radar.cloudflare.com/traffic/verified-bots, which has a nice classification. Thoughts?
What that page suggests:
Academic Research
- used only for Internet Archive, and I am not sure it's correct category. To me it would probably be a regular Crawler
Accessibility
- 3 entries, does make sense for those bots. Probably a valid category, which we can adopt.Advertising & Marketing
- based on my knowledge of how these bots work and what they do (which limited to my short time in Smartly.io) I'd say these could be treated similar to Monitoring & Analytics
category below.Aggregator
- Again, looks like a regular Crawler
to me, not sure worth it to have this as separate category.AI Crawler
- probably a valid category nowadays, although only 3 entries there. On the other hand "AI" will only imply technology used by the company, not necessarily the purpose of the bot, so regular Crawler
could still be fineFeed Fetcher
- same that what we have in 3 categoriesMonitoring & Analytics
- looks similar to our Site Monitor
Other
- has 2 items which could be considered as Webhooks
(category below)Page Preview
- essentially search bots, and some app-specific onesSearch Engine Crawler
- same as our Search bot
Search Engine Optimization
- same as our Search tools
or maybe Site Monitor
in some casesSecurity
- same as our Security Checker
and Security search bot
Social Media Marketing
- just Brandwatch in the list, which I would consider a regular crawlerWebhooks
- this feels a bit generic. I would even say that some Page review
items could be considered Webhooks
as well.Personally this is what I would do:
Assistant
category, update the bots from CloudFlare's Accessibility
botsBenchmark
-> move to Inspector
Crawler
-> keep as isFeed Fetcher
-> rename to Aggregator
Feed Parser
-> move to Aggregator
Feed Reader
-> move to Aggregator
Network Monitor
-> move to Inspector
Read-it-later Service
-> move to Crawler
Search bot
-> rename to Searcher
Search tools
-> move to Crawler
Security Checker
-> move to Inspector
Security search bot
-> move to Inspector
Service Agent
-> some can be moved to Inspector
, some to Crawler
, from a quick glanceService bot
-> I'd say Grammarly
probably can be treated as Assistant
, Vercel
- as Inspector
, ADmantX
probably, tooSite Monitor
-> move to Inspector
Social Media Agent
-> mostly image fetchers, essentially, so either Searcher
or Crawler
Validator
-> move to Inspector
So this would leave these categories:
Supporter
- bots used by various assistive technologies, including, but not limited to text-to-voice, voice-to-text, image-to-text services, translators and editorial tools.Aggregator
- bots used by tools aimed at collection and potential summarization of information from pages, including but not limited to feed readers, link or page collectors and summarization tools.Crawler
- bots not falling under other categories or related to generic or multi-purpose services.Inspector
- bots used by various tools and services aimed at monitoring, inspecting, validating and/or analyzing content or behavior of websites and users' interactions with them, including for security and/or SEO purposes.Searcher
- bots used for services related to search, including, but not limited to search engines and social networks.I also tried thinking of some acronym, but best I and GPT came up with was SCAIS, because it can be pronounced "skies". Not like we need an acronym or need these specific names, of course. But I think they are a good balance between precise and generic.
Any update would require review of all the bots. I do hope, that by the end of year I will finish going through all brands (and submit PR to correct quite a few things there) and start working on bots, and when I do I can adjust their categories as well, of course.
Variable for bots
$categories
has some ambiguous types:Feed Fetcher
,Feed Parser
,Feed Reader
- what's the difference, really?Read-it-later Service
is used only for 2 items both for 1 thing: https://getpocket.com/pocketparser_ua. At the same time description on this page clearly sayscrawling
, so should this not beCrawler
?Search tools
is used only for 1 item: http://www.shopwiki.com/w/Help:Bot. Again, description clearly states, that this is a crawler, so should this not beCrawler
?Security search bot
differ fromSecurity Checker
?Service bot
differ fromService Agent
?Search bot
fromCrawler
? I mean, crawling is done by search bots, so this seems to be the same thing.I am fine with creating PR to harmonize these things a bit, but I think this warrants a proper discussion first.