Allowing md5 based blacklists

yacy / yacy_search_server

Distributed Peer-to-Peer Web Search Engine and Intranet Search Appliance

http://yacy.net

Other

3.32k stars 424 forks source link

Allowing md5 based blacklists #445

Closed Dalethium closed 2 years ago

Dalethium commented 2 years ago

One of my main concerns with this program is indexing clearly illegal sites on tor, it should be possible to load the Ahmia blacklist into the program: https://ahmia.fi/blacklist/

Effectively it just calculates an md5 hash out of the root domain and adds it to the list, we should be able to check against md5 hashed domains.

reger24 commented 2 years ago

I'm sorry but for what should this be good for ?

the web crawler of a unmodified yacy instance running can't even access tor sides (*.onion urls) makes a blacklist with onion/tor clear names or the above md5 encryption (to hide the side name) not really a benefit for yacy.

Dalethium commented 2 years ago

That's fair -- the main reason is that my http proxy I use automatically redirects .onion to tor while keeping normal domains on a standard proxy. This means that .onion domains are indexable within my standard yacy install.

For now, I'll have my yacy install just blacklist *.onion -- if you consider this issue closed you can close it for sure!

sjehuda commented 2 years ago

That's a great idea for sharing blacklists that sharing them as plaintext may be a criminal offense. To avoid mistaken blocks, for each page blocked, YaCy will log the event against the checksum that caused it.