search

ipfs-inactive / website

This repo has been replaced by https://github.com/ipfs/ipfs-website/
https://github.com/ipfs/ipfs-website/
MIT License
183 stars 128 forks source link

Wikipedia still appears in search results, but "We would like to show you a description here but the site won't allow us." #345

Closed Mikaela closed 4 years ago

Mikaela commented 4 years ago

I was wondering whether there have been conflicts between IPFS and Privacy Badger and as I didn't see anything on PB issue tracker, I turned to a search engine and met this sight:

kuva

*https://duckduckgo.com/?q=privacy+badger+ipfs

Opening it I saw that it's from Wikipedia and checking robots.txt confirmed that it's the top one that robots aren't allowed to access. However simple Disallow isn't enough to prevent it from appearing in search results, there must also be Noindex. Google has stopped supporting noindex in 2019-09-01 according to deepcrawl.com, so I don't know what to say about this issue, but I am at least bringing it to your awareness if you hadn't already encoutnered it.

I think https://support.google.com/webmasters/answer/93710 is the answer

Mikaela commented 4 years ago

Sorry for opening this, with more sleep I think this is invalid.

  1. Something links to the PB page in IPFS Wikipedia
  2. I search for IPFS PrivacyBadger
  3. DDG offers likely result with ipfs in domain, directory and privacybadger in filename