search

privacytools / privacytools.io

🛡🛠 You are being watched. Protect your privacy against global mass surveillance.
https://www.privacyguides.org
Creative Commons Zero v1.0 Universal
3.11k stars 388 forks source link

Restore DuckDuckGo #84

Closed bakku closed 6 years ago

bakku commented 7 years ago

Hi guys,

Recently I began searching for a search engine (pun intended). Certainly I came across DuckDuckGo and searched for information since a lot of people regard it as a search engine which respects privacy.

I came across a few problems (relevant source, sadly in german: http://www.zeit.de/digital/datenschutz/2014-01/duckduckgo-startpage-ixquick-nsa) :

I suggest removing DuckDuckGo from the list and maybe taking startpage.com as a candidate. I have not found information regarding startpage which shows that it is not trust worthy regarding privacy

EDIT: I would be delighted to create a PR if others agree

ghost commented 7 years ago

I'm not sure how trustworthy they are either. Specifically because they seem more secretive about certain topics instead of open like one would expect (see here under "DuckDuckGo and Yahoo").

johnnagro commented 7 years ago

+1 value point re: american company subject to american law (national security letters, etc)

ghost commented 7 years ago

+1. IF DDG remains, StartPage should at least be added.

"DDG Privacy Policy", "We may add an affiliate code to some eCommerce sites (e.g. Amazon & eBay) that results in small commissions being paid back to DuckDuckGo when you make purchases at those sites"

ghost commented 7 years ago

@justafatalerr0r Could you elaborate further on that quote? What's the problem with adding their affiliate tag to links in search results?

privacytoolsIO commented 7 years ago

@bakku Good point. I have no problem with removing DDG. Should it be replaced with StartPage?

ghost commented 7 years ago

Could someone take a look at this discussion? As far as I can see these requests should be blocked by most adblockers but it still made me think if SP is as trustworthy as they'd like you to believe. On the other hand they could probably hide this data collection from the user if it really were problematic. (?)

bakku commented 7 years ago

@IDKwhattoputhere Interesting. I will have a deeper look at this and come back here when I have some results

privacytoolsIO commented 7 years ago

I've replaced DDG with StartPage for now.

yegg commented 7 years ago

Hi, I'm a bit confused by this assessment. We (at DuckDuckGo) believe we are as private as you can get in terms of search. Responding to some things in the thread:

While we are headquartered in the US, our situation is different than other companies because we do not collect any personal information at all. US laws in this area are generally are about requesting existing business records of some kind (metadata or underlying content), as opposed to creating significant new source code to surveil. That's why the Apple case was such a big deal. As a result, services where you actually store personal information are in very different situations than those where no personal information is stored (like us).

Additionally, if you're worried about US organizations like the NSA in particular, you should note that inside the US they have legal restrictions (they cannot spy on US citizens) that prevent them from taking certain actions, but outside the US they have no such legal restrictions, and are therefore free to operate clandestine operations without any similar threat of legal recourse. In other words, any server or network outside the US that is an interesting target is much easier for the NSA to compromise.

With regards to Amazon, all traffic sent to DuckDuckGo is encrypted (A+ at SSL Labs including PFS - https://www.ssllabs.com/ssltest/analyze.html?d=duckduckgo.com), and that encryption protects your query in transit to our servers, which are solely controlled by us. Additionally, all sites need to be hosted somewhere, and as I mentioned above, those hosted outside the US operate under less legal protection from US surveillance organizations. DuckDuckGo also has servers around the world, and if you are in Europe you will be connected to our European servers.

With regards to Yahoo, I've reached out to the author of that article and he is presently revising it. We have never sent any personal information to Yahoo or any other partner, and we of course do not collect any ourselves. Those pages the mentioned article references were removed because our implementation actually did change on the backend, and they are no longer relevant. Similar to needed to being hosted anywhere, any private search engine needs to work with similar partners to get a full set of results.

I'm happy to answer any questions.

ghost commented 7 years ago

(they cannot spy on US citizens)

Why do you say that when it doesn't appear to be true?

In theory, the NSA is forbidden from spying on U.S. citizens. But in practice, a secret 2015 court ruling unsealed this week reveals that warrantless spying has been approved by the Foreign Intelligence Surveillance Courts for general investigations in the U.S. Furthermore, the NSA says it wants to share access to communications databases with other domestic law enforcement agencies, including the FBI.

More: https://www.eff.org/nsa-spying http://www.pbs.org/wgbh/frontline/article/with-or-without-the-patriot-act-heres-how-the-nsa-can-still-spy-on-americans/ http://www.huffingtonpost.com/2014/12/26/nsa-spying-report_n_6382572.html http://www.dailydot.com/via/edward-snowden-nsa-americans-fourth-amendment/ http://www.vice.com/read/the-fbi-wants-to-wiretap-every-us-citizen-online http://www.newyorker.com/news/amy-davidson/how-many-americans-does-the-n-s-a-spy-on-a-lot-of-them

Also you're talking about citizens when DDG is a company. Am I misunderstanding something here?

yegg commented 7 years ago

The central point around the NSA is that if you're worried about the NSA, you are arguably less protected outside the US where they have absolutely no restrictions on their actions. Additionally, US surveillance laws are generally about turning over existing business records with personal information, and DuckDuckGo has none.

The bigger point though is that the US is just one country, and as privacytools.io notes, many countries share intelligence and have their own surveillance operations. Really all relevant countries' legal situations need to be analyzed to get a full threat assessment on a particular attack vector. That's why to me it is an important distinction if there are services that can operate without collecting any personal information at all, which is the case in search, and what we do at DuckDuckGo.

bakku commented 7 years ago

Hello @yegg, it's really great that the CEO of DuckDuckGo joins this discussion, thanks!

First of all we could now argue about which citizens of a country are more under surveillance and which are less but I think we can presume that we are all under surveillance, no matter where but this is not really the topic here.

While we are headquartered in the US, our situation is different than other companies because we do not collect any personal information at all.

I read your privacy policy and your philosophy is really great. You furthermore stated that it does not generally happen that the national agencies request to implement new code for surveillance. But history has shown that it can happen and it also can happen in a way in which you are not allowed to tell your users.

Now we could argue like you did that we should analyse every countries legal situation since maybe this might happen somewhere else as well and you are right. That's why any person who has found out legal information about a country can post an issue to privacytools.io to further improve the site. It's just that we already have experienced this with the US.

Additionally, all sites need to be hosted somewhere, and as I mentioned above, those hosted outside the US operate under less legal protection from US surveillance organizations

For me I can't see a problem where Amazon could collect data from DuckDuckGo and their servers since you don't collect it but also I can't imagine the power they have actually. The problem is generally that Amazon can not be trusted at all since they sold themselves to the CIA. I know that AWS is handy but why does a company which has such a great privacy philosophy then use the services of a company whose opinion on this is totally the opposite.

Kiyuubi commented 7 years ago

Some stuff on DDG: https://8ch.net/tech/ddg.html

yegg commented 7 years ago

Thank you for your recommendation of our privacy policy. We try to set an example because we believe in services putting forth straightforward privacy explanations that spell out clearly the benefits you get as a consumer for giving up particular pieces of personal information. In our case of course, we collect no personal information, but in the general case we believe services should collect the minimum possible.

Our vision is to raise the standard of trust online and we do that through our donations to privacy organizations (https://duck.co/blog/post/303/2016-foss-donations-announcement) and our mission to be the world's most trusted search engine. If we believe we could do something to better protect our users' privacy, we would do it, and are more than willing to entertain suggestions.

The argument put forth here seems to be that anything touching the US or Amazon is less trustworthy than anything that doesn't touch them. I know this is not the case, and that it is a much more nuanced reality. And in our particular case, it is actually more clear cut since we do not collect any personal information.

I thought that you perceived these nuances since you already recommend many organizations with these properties, but if you're going on this essentially ontological bogeyman argument, there isn't really any more I can say here.

The bottom line is if you'd like to recommend a private search engine, I whole heartedly believe you can do no better than DuckDuckGo. I believe everybody should adopt a private search engine, and so I do not engage in debates maligning other private search engines, but I know that if you analyze completely the full threat assessments in reality, you will find DuckDuckGo to be just as private, if not more, than any other provider.

RealOrangeOne commented 7 years ago

After having read all this, I am a little more sceptical on using DDG, however i'm still going to use it. It's not perfect, but comparing features, security, and how dodgy it looks, it's my favourite!

Yes it's based in the US, but being outside the US, provided I connect to EU servers, i really dont care. Yes, amazon are known to share data with government bodies, but depending on how their network is setup on AWS (information that obviously isnt public), it's possible it's not all bad.

My largest complaint is with the afffiliate links from search pages. subtly injecting this into URLs worries me, especially seeing as there's no way to disable this. I'd much prefer being served an ad based on my search query (provided it was done securely / anonymously), than having affiliate links links. I'd happily take this as a choice in the settings between ads and affiliate links.

yegg commented 7 years ago

@RealOrangeOne thank you. With regards to affiliate links, there are no privacy issues with them whatsoever. The only programs we use are Amazon and eBay because those are the only two programs I know of that can used completely anonymously. From https://duck.co/help/company/advertising-and-affiliates:

This mechanism operates anonymously and there is no personally identifiable information exchanged between us and Amazon or eBay. These links are regular organic links (like any other link in our results) and these programs do not influence our ranking or relevancy functions in any way. That is, they are not advertising like paid placements or paid inclusions, and we only generate revenue from them if you ultimately find them relevant enough to end up purchasing an item. For more details, check out our privacy policy.

With regards to EU servers, as said above, we do operate EU servers and so you should be interacting with them directly by default if you are in the EU. For people in the US, using EU servers doesn't really get you anything since your traffic has to physically flow through the US, and we do not store personal information in any case.

aloisdg commented 7 years ago

Long time DDG user. I am also using Qwant (mostly for french stuff):

Qwant's philosophy is based on two principles: no user tracking and no filter bubble. We do our best to respect the privacy of our online visitors while ensuring a secure environment and relevant results. Here are our commitments for the user’s data protection : If you wish to register or log on your Qwant account, or to send us a request via our contact form, we may ask you to disclose personal data. Thus, you are entitled to protection under the European data protection regulation. This Privacy Policy aims to present our ethical positioning with regard to the collection and processing of data: we guarantee not to sell or disclose the user’s data in any way, especially for commercial purposes.

source

What do you think of it?

I am for keeping DDG but with a caveat and a link to their privacy policy. We cant trust promises, but they are better than nothing.

jaredStef commented 7 years ago

I vote keep DDG

moonmehta commented 7 years ago

I think DDG should be put back. It's fine to put StartPage and Qwant alongside too. All of the three are private enough and I think we should ultimately let the user decide.

hovancik commented 7 years ago

SIte says about another services/products: 

Operating outside the USA or other Five Eyes countries. 
More: Avoid all US and UK based services.

Should apply on all services/products.

moonmehta commented 7 years ago

Also, consider that StartPage is really a meta search engine ultimately. That means that it ultimately has a dependency on Google's search results. It doesn't affect our privacy directly but it does mean that the problem remains fundamentally unresolved. DuckDuckGo on the other hand is relatively independent and therefore represents a somewhat cleaner alternative.

ghost commented 7 years ago

The important questions for me now are:

I personally use DDG and would like DDG to be on the list but it's up to the project and what it's intentions are.

bakku commented 7 years ago

You are totally right @xdtnguyenx. After having a thought my opinion is that the most recommended way as far as possible would be what you called "extreme privacy" but there should be a place for alternatives as well which might not align perfectly with all privacy recommendations since otherwise this projects recommendations will just be useful for people who are willing to take huge sacrifices. So in this case DuckDuckGo would be a totally valid choice for a search engine and I would take my initial statement back and at least have it shown on the "Worth mentioning" section.

moonmehta commented 7 years ago

I will copy-paste a comment here from the reddit discussion that is taking place about this.

There seems to be a lot of "boogyman" statements here and on the discussion. Heres the thing, you either trust or you dont. Now i know a bit about how search engines work, and hiw netwirks communicate. The thing about ddg that i like is that you get a lot of good features that other private search engines dont offer.

  • turning your searches to GET requests or POST requests. This changes packet headers and shares less or more info about you as a user.
  • you can use ddg without cookies
  • no user accounts
  • search redirect ability so other sites dont get SentFrom or LinkedFrom information about you in the http packet header.
  • currently not required to retain data
  • legally protected from turning the project into a surveillance tool.

These are all things we know. These are able to be validated. We can play the "what if they are lying" game to the end of time... But we could make great use of occams razor here, and make the least amount of assumptions and look at real information instead.
We cant have discussion when anyone is a government agent, anything could have an open backdoor, anyone could be lying.
Instead look at what you can quantify, what you can verify, and what you can trust rather than blanket assumptions based on fear.
After looking into the service from my side many times, i can say that i trust ddg with my daily searching activity needs.

Link to original comment - https://www.reddit.com/r/privacy/comments/5j5pwy/interesting_discussion_with_the_ceo_of_duckduckgo/dbe67ld/

I think it makes sense to include DDG considering that they don't have any data about the user in the first place + all of the above.

ghost commented 7 years ago

@yegg

The central point around the NSA is that if you're worried about the NSA, you are arguably less protected outside the US where they have absolutely no restrictions on their actions.

That really depends though, doesn't it? Not only on if they don't find a way around restrictions or get green-lit by a secret court but also on how closely they're watching for example. It's not like the NSA spies on the US just a little and on every other country a lot (see, found this interesting too). Additionally it's easiest for them to send out an NSL which they can only do to US companies. If they had a better way they wouldn't be using NSLs in the first place.

yegg commented 7 years ago

@IDKwhattoputhere there is a good discussion on the reddit thread referenced above on how NSLs do not apply to DuckDuckGo in any straightforward manner because we do not collect any personal information.

Even though NSLs can be issued without a judge's signature and can come with a gag order, they are just a legal tool that can be used to extract certain types information (such as subscriber information and maybe a little bit of transactional information) that a service provider already has stored on their servers. NSLs can't be used to force a service provider to start collecting data or build a backdoor. https://www.youtube.com/watch?v=YN_qVqgRlx4&feature=youtu.be&t=20m16s

ghost commented 7 years ago

I don't think that comment is entirely correct. Specifically the backdoor part:

An example of this is Lavabit – a discontinued secure email service created by Ladar Levison. The FBI requested Snowden’s records after finding out that he used the service. Since Lavabit did not keep logs and email content was stored encrypted, the FBI served a subpoena (with a gag order) for the service’s SSL keys. Having the SSL keys would allow them to access communications (both metadata and unencrypted content) in real time for all of Lavabit’s customers, not just Snowden's.

yegg commented 7 years ago

@IDKwhattoputhere the two videos referenced in that thread (https://www.youtube.com/watch?v=YN_qVqgRlx4&feature=youtu.be&t=20m16s and https://www.youtube.com/watch?v=PX2RjJAfTYg) do a good job of explaining much of the nuance around the US surveillance legal situation, including backdoors. The part in the second video that specifically addresses this point is at https://www.youtube.com/watch?v=PX2RjJAfTYg&t=27m12s, though the whole video is worth watching.

"There is no obligation to build in backdoors. ... End-to-end encryption is legal. Period."

Lavabit was in a fundamentally different situation than DuckDuckGo because they collect personal information, as any centralized email service has to. That's why Silent Circle preemptively shut down their email service but kept their other end-to-end encrypted services up.

Silent Mail has thus always been something of a quandary for us. Email that uses standard Internet protocols cannot have the same security guarantees that real-time communications has. There are far too many leaks of information and metadata intrinsically in the email protocols themselves. Email as we know it with SMTP, POP3, and IMAP cannot be secure...

Silent Phone and Silent Text, along with their cousin Silent Eyes are end-to-end secure. We don’t have the encrypted data and we don’t collect metadata about your conversations. They’re continuing as they have been. We are still working on innovative ways to do truly secure communications. Silent Mail was a good idea at the time, and that time is past.

In a similar statement to Techcrunch:

[Users] are completely secure and clean on Silent Phone, Silent Text and Silent Eyes, but email is broken because govt can force us to turn over what we have. So to protect everyone and to drive them to use the other three peer to peer products- we made the decision to do this before men on [SIC] suits show up. Now- they are completely shut down- nothing they can get from us or try and force from us- we literally have nothing anywhere.

That's the difference between services that collect personal information, metadata or otherwise, vs services (like DuckDuckGo) that collect nothing.

ghost commented 7 years ago

I haven't read the entire discussion, but:

So, what's the issue? Being afraid that DDG may be forced to change their policy (log all search queries and IP addresses) is like being afraid of using SSL/TLS, because CA's can track your Google searches.

ghost commented 7 years ago

@Shifterovich: @bakku raised the point that hosting on Amazon is the issue. @RealOrangeOne said that it's possible it's not all bad depending on how they set up their servers which I would like them to expand on. The way I see it is that Amazon has access to the servers and will compromise them if asked to.

ghost commented 7 years ago

DDG will compromise themselves if asked to. NSLs. We can't just consider the cons, there is a ton of pros too. What are the pros of recommending some other search engine?

ghost commented 7 years ago

The Lavabit case showed that the US government can request a service provider to hand over its SSL/TLS keys. These are, after all, something that you would expect a service provider to have access to. Normally, an adversary that can gain access to a web service’s SSL/TLS keys would be able to impersonate the server going forward. If the web service’s user hadn’t used Tor when accessing the web service, and the adversary had logged the ciphertexts of previous user-server connections from a position between the user and the server, then the adversary would also be able to use the SSL/TLS keys to decrypt those ciphertexts and reconstruct what the server would have contained if it had logged all previous user-server interactions (even if the web service’s server hadn’t actually done so).

@yegg: I’m interested in knowing what the consequences would be for non-Tor DDG users if someone gained access to your SSL/TLS keys, assuming that the adversary had already collected the ciphertexts of most if not all previous user-DDG interactions. Do you think that the adversary would be able to reconstruct most if not all of what your servers would have contained if your servers had logged all previous user-DDG interactions? What would DDG do if the US government requested that you hand over your SSL/TLS keys? If you did hand over your SSL/TLS keys, would you revoke your old keys, generate a new set of keys, and then continue operating as before?

Some related questions: Does DDG have separate SSL/TLS keys for each jurisdiction in which you operate? If so, can you guarantee that the keys in one jurisdiction are outside of the reach of another jurisdiction in which you operate? As an example, Microsoft has made Deutsche Telekom the "custodian" for their web-based services in Germany.

Edit: Added idea of revoking old keys to a question.

ghost commented 7 years ago

VPN providers being based in the US is an issue, search engine - not really.

Never trust any company with your privacy, always encrypt.

emily-pesce commented 7 years ago

Do you think that the adversary would be able to reconstruct most if not all of what your servers would have contained if your servers had logged all previous user-DDG interactions?

So the attack you're worried about is an adversary somehow MITM/collecting and storing all encrypted requests to a service and then later somehow getting their hands on that service's SSL/TLS keys and using them to unencrypt/decrypt and thereby reconstruct a user's activity?

I mean, sure that's technically plausible, but by this logic DDG being based within the US is advantageous because no government adversary should be collecting such requests within US borders. Right?

But, I think you've already answered the question you're alluding to - if this is the use case you're most concerned about with a service like DDG then Tor or something similar is the answer. Otherwise, I think DDG is miles better than Google/Bing/similar and is a reasonable recommendation.

ghost commented 7 years ago

^ also, how is that related to US?

ghost commented 7 years ago

I suggest adding DDG with a note that it's based in the US.

ghost commented 7 years ago

@Shifterovich

VPN providers being based in the US is an issue, search engine - not really.

Could you elaborate on that? I think your reasoning behind this could add to the conversation.

@michael-pesce

I mean, sure that's technically plausible, but by this logic DDG being based within the US is advantageous because no government adversary should be collecting such requests within US borders. Right?

Wouldn't this only apply to US citizens since the traffic otherwise would be foreign traffic and not be covered by those protections?

ghost commented 7 years ago

@IDKwhattoputhere VPN can capture your entire internet activity, a search engine can capture your searches.

emily-pesce commented 7 years ago

@IDKwhattoputhere

Wouldn't this only apply to US citizens since the traffic otherwise would be foreign traffic and not be covered by those protections?

Yes, sorry, was speaking from the perspective of a US citizen.

moonmehta commented 7 years ago

@xdtnguyenx sums up the situation correctly that we need to decide whether we are going for the extreme or not. In the extreme case, all the countries with five eyes should be on our radar. And the extended 9 countries as well. @bakku agreed to that in this context, DDG seems fair enough to get a position in the recommendations. Considering DDG's no data retention, I fail to see the problem really. If we are really going for the extreme end, then why is the discussion even happening on Github? Why is the project hosted on Github in the first place?

ghost commented 7 years ago

In the extreme case, we shouldn't be using TLS with CA's.

In the extreme case, all the countries with five eyes should be on our radar. And the extended 9 countries as well.

In the extreme case, all countries should be on our radar.

kewde commented 7 years ago

I think it should be noted that this discussion should focus on the whether or not a search engine within the general context is the most private and not enter the field of extremes.

SSL keys can be handed over, that's indeed a weakness but it is one shared among all search engines. I think it a better criteria should be what practices do the search engines apply and to what degree are they transparent about it? As mentioned above, DDG does a very good job at that.

I also believe that the NSA argument is weak and should not play as big of a part in the threat model as some of you want it to be. If they are out to get you, they will get you. Nobody is safe from their claws.

DDG has servers all over the world, which weakens the traffic-logged-when-entering-US argument.

DDG also provides an onion link, which any privacy conscious individual should use. They even https over Tor iirc (a subtle yet nice addition).

I tend to agree that it should be in the top 3, put DDG back on their spot.

ghost commented 7 years ago

I say we should test all these search engines - how they search. A good search engine based in the US is better than a shitty search engine based in a better location. To decide what should be first, second, third, and in the Worth Mentioning section.

bakku commented 7 years ago

In the past months I have used searx.me, Startpage and DuckDuckGo each for a certain amount of time. I would like to present a small (personal) comparison between those three.

  1. searx.me
  1. Startpage
  1. Duckduckgo

Would like to hear about other experiences 🙂

Atavic commented 7 years ago

YaCy is a free search engine that allows a local and decentralized Web Search.

The questions about NSA are useless, you can't avoid the monitoring done at international backbone level. What should really be of concern to a privacy aware user?

Tracking, Fingerprinting and behind-the-scene techinques as Cookie Syncing.

ghost commented 7 years ago

I looked for an option to only use google search results with searx.me but I didn't find any...

This or use !go example.

bakku commented 7 years ago

You're the boss. As my defense, I based my previous statement on searching for a solution in forums, github & co, not on my own 😬

I take back what I said.

GreenLunar commented 7 years ago

DDG should pay you to include them!

I am for removing DDG for two reasons:

Keep DDG only if they pay you a monthly fee.

bakku commented 7 years ago

@GreenLunar I think that respecting user privacy today is a business model because users are willing to pay for that, so I think it can be taken in consideration in a company that wants to make profit. Furthermore, many products other than open source projects would fall in the same category as DDG then as well like VPN providers or mail providers

GreenLunar commented 7 years ago

There is no such thing for end-users, in any case, as they almost always press OK without reading and investigating anything; and maybe tomorrow the model of DDG would not be privacy-driven.

I still think that DDG should pay a fee in order to be listed in privacytools.io.