search

AdguardTeam / AdGuardHome

Network-wide ads & trackers blocking DNS server
https://adguard.com/adguard-home.html
GNU General Public License v3.0
24.82k stars 1.79k forks source link

Solution to block worldwide services under particular categories #2230

Open Aikatsui opened 3 years ago

Aikatsui commented 3 years ago

Product: AdGuardHome is a FOSS and On-premises (you own your data) software then i considered to mention resources from other products to get better idea for further development

Why this feature?

  1. Definitely that'll be parents/employers most loved feature in this product. It means, you don't have to block steam or tiktok then struggle with mind, what'll my kids use next, they usually find something that parents never have heard for life. (ex: there're lot of tiktok alternatives similar chingari, dubsmash, triller, funimate etc)
  2. Continuously add new services to 'Block services' page just make it too complicated and when it become large list then you cannot recognize necessary services (or type of services) to block easily
  3. You can block worldwide services under certain category. it'll provide most effective coverage (ex: block chat apps category then whitelist 'Signal' domains for DNS server users to communicate)
  4. 'Blocked services' component suitable for dominant internet companies and adguardhome is a multi-platform software, already has millions of users (ex: 10M+ only docker downloads) then adding regional services leads (ex: social media),
    • to be a one of source that bring foreigners to regional platforms and expand numerous disappointing troubles.
    • to be a advertising/new users generate source to those sites or apps
  5. Missing something? You can check query log (or with possible enhanceable mode https://github.com/AdguardTeam/AdGuardHome/issues/1164) and report to adguard https://github.com/AdguardTeam/AdGuardHome/issues/1133

After checked 3 cloud based providers then below contain appropriate categories(38) list

Parental Control Enhanced parental control (or adult content filtering) (It include personals & dating, Intimate apparel & Swimwear and other adult or adult mixed contents) It seems adguard family dns doesn't block dating sites and similar.

Entertainment Media Sharing Streaming Media Entertainment

Computers Proxies, VPNs, P2P & anonymizers #101 Online Storage & File sharing Search Engines & Portals Visual search Downloads Text-Audio only Content Servers Torrents

Personal web content & social networking Chats Comics, Humor & Jokes Forums & Newsgroups Hosted personal pages Instant messaging Moderated forums Blogs & Wikis Parked sites Social networking

Games Gaming Game playing & Game media Game-cartoon violence

Illegal Activities/Cheating Tasteless & Offensive Illegal/ Piracy software Weapons

Health & Medical Fashion & Beauty Health & Medicine

News News

Recreation/Hobbies Food & Dinning Hobbies & Recreation Sports

Shopping Shopping

Travel Travel

Misc Real estate & property businesses

Drugs / Alcohol / Tobacco use or abuse Alcohol & Tobacco Illegal drugs, use and educations

Resources

Comodo dome shield provide 80+ categories Adult/Sexual* - Nudity* - Pornography* - Adult contents - Intimate apparel & Swimwear - Personals & Dating* (Note: some words above aren't suitable to add to product in anyway due moral reasons even for adults and children may use/facilitate this software as well) Arts & Entertainment - Media Sharing - Streaming Media - Entertainment Business - Advocacy-NGO - Auctions & Classifieds - Business - Finance & Investments - Job search & Career development - Marketing & Merchandising - Stock Trading Computers - Proxies - Online Storage - Search Engines & Portals - Visual search - Downloads - Information security - Online services - Software-Hardware - Advertisements & Popups - Computing & Technology - Peer-to-Peer - Text-Audio only - Content Server Cultures/Society - Arts & Society & Culture Drugs / Alcohol / Tobacco use or abuse - Alcohol & Tobacco Illegal drugs/Alcohol/Tobacco use or abuse education - Illegal drugs Education - Education & Reference Games - Gaming - Game playing & Game media - Game-cartoon violence Illegal Activities/Cheating - Tasteless & Offensive - Illegal software Health & Medical - Fashion & Beauty - Health & Medicine News - News Recreation/Hobbies - Weapons - Food & Dinning - Hobbies & Recreation - Motor vehicles Shopping - Shopping Personal web content & social networking - Mobile communication - Professional networking - Public Information - Chat - Comics & Humor & Jokes - Forums & Newsgroups - Hosted personal pages - Instant messaging - Web applications - Kids sites - Moderated forums - Blogs & Wikis - Parked sites - Social networking Sports - Sports Travel - Travel Misc - Real estate - Technical Information - Political issues - Religion - Government & Legal
From NextDNS - Porn - Gambling - Dating - Piracy - Social network
From Cleanbrowsing - Adult & Pornography - Adult mixed contents - Ads & Tracking - Torrents - P2P & File sharing - Proxy & VPNs - Gambling - Dating - Search engines - Weapons - Malicious - Gaming - Social networks

97213931-e36d9200-17f4-11eb-92b9-4f0c5ad3d26e

jeee commented 3 years ago

Nice. Some additional ideas

Aikatsui commented 3 years ago

Comment Hidden - Off-topic for this feature request

  • block a category it then maybe be able to enable a certain service (say signal and block all messaging)

That's possible already then, you can create a DNS allowlist or custom filtering rules. (or it might be possible to enhance, custom filter list editor (web-interface) #1450 to fetch necessary domains from category automatically when type 'signal' and help user to create the allowlist(whitelist) immediately without having to spend time for check query log)

  • blocking the tiktok category but enable only tiktok for a certain time of day

There are two options (FRs) for this,

  • give time access for each user (for example 14 year old has access to 30 minutes of tiktok while the 8 year old has none)

It require below features

It's easy to say, necessary feature requests already exists that helps to accomplish your requirements, @jeee but adguard may feels confuse though.....

DandelionSprout commented 3 years ago

I am of the viewpoint that relying on toggle buttons to block these things are not ideal and also not even properly FOSS-philosophy, since the use of standard filterlists would be preferable and require less effort to check the contents of.

In particular, going to https://filterlists.com/ and searching shalla list, shows raw-domains lists for half or more of the topics listed in the OP.

neotrak commented 3 years ago

When will adguard team add this feature to home? eta? It's difficult for us to check-block manually. we dont know what'll other person or kid search on internet, sometimes people decide to use sites heard from friends, media sources or find alternatives. For dns server manager, it can be hundred and thousands of online games, social apps and other sites.. i have experienced it myself with adguardhome and i'm not satisfied about blocked services feature, already gave up. therefore we are using paid service provide by a local company for my institute and our home

ameshkov commented 3 years ago

It's not planned on any milestone yet.

The feature request is rather complicated, and the problem is websites' categorization data:

  1. It's unclear where to get it.
  2. If this data is local, we'll have to come up with an update procedure (which is rather tiresome)
  3. Alternatively, we'll have to come up with a web service that will supply this data. Also, not a simple thing to do.
DandelionSprout commented 3 years ago

It keeps surprising me quite a bit that A) a significant amount of AdGuard Home's userbase use AGH for workplace censorship instead of adblocking, and B) that the 18 👍's to the OP ignored https://github.com/AdguardTeam/AdGuardHome/issues/2230#issuecomment-723939361.

To block Flash/HTML5 game sites, Pokémon GO, and a small amount of gacha game resources, these lists should at least be a good start:

To block sites where one can gamble with money:

neotrak commented 3 years ago
  1. If this data is local, we'll have to come up with an update procedure (which is rather tiresome)

This way, users can get updates daily and store in a DB file. comodoshield is slow for us. it shall not ideal unless adguard team run many servers. our local provider work as a middle-man between ISP and customer. they specially do data mining to provide number of services. It's like ADGuard DNS analyze queries history to improve ad-blocking but it's much more less than what that company does. Yeah, it's unclear about a good way to get this data

@DandelionSprout adguard home has censor features. no fever, it does more than what we hope in simple centralized environment and take control over entire network.

Aikatsui commented 3 years ago

@ameshkov It's not planned on any milestone yet. The feature request is rather complicated, and the problem is websites' categorization data: 1. It's unclear where to get it.

See below solutions, maybe you can pick affordable services (one or combine multiple) and some companies interest to provide their services for free or under discounts to foss products but do no integrate them directly, we do not like it then adguard has to do separate task before serve it. These companies have threat and adult content filtering facilities, it's possible to use them for improve adguard default DNS and family DNS

However, I noticed several sites pricing too high then as i said above, it's better to get a service for free or find free services.

ameshkov commented 3 years ago

@Aikatsui ideally, we'd prefer an open-source categorization solution for AdGuard Home integration (if that's possible).

Aikatsui commented 3 years ago

Meantime found that many platforms has this facility since long time then adguard cannot delay to implement easily ;)

Comodo dome shield, Cleanbrowsing, NextDNS, SafeDNS, SmoothWall, FortiGuard, iBoss, Barracuda, Sophos Web Filtering, Titan Content Filtering, OpenDNS, Netnany, FlashStart, DNSFilter, Mimecast Web Security, McAfee Web Gateway, Clouduss web categories, Forcepoint, Codeproof, ScoutDNS, GoGuardian and more Dnssafety Diladele, NxFilter

I noticed another interesting feature has in many products, real time analyze content, construction, context, URLs then categorize unknown domains. it's possible to build as offline as aggressive feature (otherwise cloud based(ML) solution for all AG products)

@Aikatsui ideally, we'd prefer an open-source categorization solution for AdGuard Home integration (if that's possible).

If there something mention about open-source then found these but it is less likely usable for the purpose as see from outside.

Other resources (Free)

OperaART commented 3 years ago

Does virustotal db work for this?

OperaART commented 3 years ago

image image

naDevi commented 3 years ago

Please take priority consideration on this feature request @adguardteam Two participants have provided their opinion for shallalist. It's free and offline database. can you guys add it?

https://aws.amazon.com/awis

https://aws.amazon.com/alexa/ Alexa web information service is very cheap. personally i believe they have the best database and you can keep in cache every queried domain category some years before re-update. this is a nice feature for all adguard products (@Aikatsui :star2:) 1,001 to 1,000,000 requests/ month - $0.00045 per request Over 1,000,000 requests/ month - $0.0003 per request

Edit:

It's better to dicuss with amazon tech support and make clear everything before add funds otherwise $450 will be evaporate within second 😅

ameshkov commented 3 years ago

Interesting, thank you for the info!

Meantime, I'll see if we can use Shalla List.

DandelionSprout commented 3 years ago

Shalla normally has licences that discourage use by companies (https://www.shallalist.de/), and it's not 100% obvious if the Shalla team is aware of the GitHub re-uploads of their lists (https://github.com/cbuijs/shallalist/).

If that was to be sorted out, I see no problems with using at least some of Shalla's lists.

ameshkov commented 3 years ago

I've contacted them to ask if it's possible to use it in AGH

naDevi commented 3 years ago

@ameshkov Any news?

ameshkov commented 3 years ago

Surprisingly, no response yet.

naDevi commented 3 years ago

Surprisingly, no response yet.

I called to the phone number available on this page (2 days) http://www.shalla.de/impressum.html but they didn't answer.

Aikatsui commented 3 years ago

I noticed mentioned one list offer under creative common license https://dsi.ut-capitole.fr/blacklists/index_en.php recently The maintainer "fabrice prigent" interest about adguardhome, may be already following this project

@ameshkov It has around 60 categories, update daily and the list maintain since 25yrs. It's possible to download via http, ftp, rsync If it good then i suggest to assign this FR to milestone because upto 20 competitors have this feature already.

ameshkov commented 3 years ago

Thank you for the suggestion! I am still not ready to assign a milestone, though, I'd only be able to do that after the redesign is completed.

ghost commented 3 years ago

+1

kyith0 commented 3 years ago

yup, this will be very useful :+1:

piyushgarg commented 7 months ago

Either categorize them or put the labels like (chatting, video streaming, gaming, betting, porn, shopping, photo sharing etc) this on the services. image