Domains visited get leaked to DDG servers

[Tritonio]

https://github.com/duckduckgo/Android/blob/ed91c9e551d2a9e1559199f110bd94c076784c71/app/src/main/java/com/duckduckgo/app/global/UriExtension.kt#L60

This seems to be leaking all(?) the domains that users visit to your servers.

[tagawa]

Hi @Tritonio and thanks for your feedback. The purpose of the request you observed is to retrieve a website's favicon so that it can be displayed in certain places within the app or on the results page. We use an internal favicon service because it can be complicated to locate a favicon for a website. They can be stored in a variety of locations and in a variety of formats. The service understands these edge cases and simplifies retrieval within our apps and our search engine.

At DuckDuckGo, we do not collect or share personal information. That's our privacy policy in a nutshell. For more detailed information on that, you can checkout our privacy policy at https://DuckDuckGo.com/privacy. The favicon service, as with all our services, adheres to this privacy policy in that the requests are anonymous and do not collect or share any personal information.

If you have further questions, please let me know.

[stefan01]

Hi @tagawa, i do trust DDG not using the data in a bad manner, but still i don't get your decision. The best privacy products are made from companies you don't need to trust using your data in a good way, because they collect no data. So for a privacy product it is really hard for me to understand why you chose a minimal performance increase (?) over a critical privacy issue (in my opinion). I think with this decision you are risking a major privacy distrust.

[donfuxx]

I understand that there are different favicon standards and that in some cases it can be difficult to locate it. I believe that those are edge cases though and for a vast majority of websites a simple host/favicon.ico should work. My suggestion for an increased privacy here is:

1. Attempt to load favicon directly from website by simply appending /favicon.ico to hostname (should be the most easy and most common case anyway)
2. In case of no success fallback to use omniscient ddg favicon endpoint

[stefan01]

Just to understand: Why can't the same algorithm, which is being used in the DDG endpoint, be utilized in the app? If I understand correctly the only reasons are slightly increase performance, reduced network traffic and easier implementation. Is that correct?

[masstransithonchkrow]

Hi there! If DDG reviews this, please use the header data on my web pages to determine what the favicon is. I do not have a default favicon for my site. My DeskThemePacks use PNGs for favicons based on the currently viewed theme's normal cursor.

By looking for favicon.ico where it doesn't exist, it affects both my site's performance and SEO.

[timmc]

If it was just search results, I wouldn't care.

But this appears to be in use for rendering tabs that you're visiting, regardless of how you got there:

https://github.com/duckduckgo/Android/blob/db728523240e377277deb88af71d751eb8bdb5dd/app/src/main/java/com/duckduckgo/app/tabs/ui/TabRendererExtension.kt#L44

(ETA: Or am I misunderstanding how this app is used? I'm not a smartphone user, and only use DDG in a regular browser.)

[tonyxu-io]

It’s shocking to see how DDG staff responded to this issue. Doesn’t seem to care much about user privacy concerns.

[jeevank]

@CDRussell @tagawa This needs to be reopened and looked into. This is a serious privacy issue regardless of your "privacy policy"

[sheerun]

This made me really doubt privacy of DuckDuckGo. How it can be private and secure if you hire security experts who allow it to happen since 2019? "it can be displayed in certain places within the app or on the results page"

[davewood]

Take the code from Firefox iOS or Android-components. We spent a lot of time on these and it is all on device.

https://github.com/mozilla-mobile/android-components

https://github.com/mozilla-mobile/Firefox-iOS 

[https://news.ycombinator.com/item?id=23708166]

[javabeanz]

privacy should be the whole raison d'etre of DDG. time for a fork DuckDucGone ?

[scabros]

I don't know why all the fuss... You all know that if you are using his app, you already "trust" that his search service is respecting your privacy, right? C'mon, they are being transparent, as usual.

[tonyxu-io]

I don't know why all the fuss... You all know that if you are using his app, you already "trust" that his search service is respecting your privacy, right? C'mon, they are being transparent, as usual.

@scabros - Please checkout the reply above: https://github.com/duckduckgo/Android/issues/527#issuecomment-652721495 This is not about using their search service, we all know search provider know exactly what you are searching for. But this particular issue is about using the browser, it's when user directly visit the website from browser address bar or reference link.

[svenssonaxel]

So essentially, the trade-off is between performance and auditability of privacy. I predict that the user segment DDG appeals to will overwhelmingly prefer the latter.

Yes, we already trust DDG, but only because we have to trust someone and others have proved to be untrustworthy. The issue isn't about whether the user trusts DDG, it's about minimizing the need for trust and maximizing the ability to verify privacy. Please consider reopening this issue.

[eighthave]

I think this would make this app qualify in F-Droid for the Tracking anti-feature, there is no good reason why every website URL should be sent to DDG servers. Regardless of intent, it does key bits of data, and there are better technical solutions to finding the favicon.

[solomoncaygnuyou]

I believe reopening and rectifying this design choice would go a long way to preserving the trust that DDG has worked hard to earn in the privacy-oriented community. Even if users trust that DDG is not using this data in any undesirable way, it would provide more peace of mind to users that the data not be sent in the first place.

I am strongly in favor of changing this functionality to no longer leak visited domains.

[holderbaum]

A long time DDG user and advocate here.

First and foremost, thank you for building the most awesome search engine there is right now! :slightly_smiling_face:

I must say that I am pretty appalled by this implementation. Yes, I trust DDG with my search data since I have to do searches and DDG always seemed like they actually care about privacy. But at the same time, a privacy aware browser should never send close to all the user activity to it's server, even if it is "only" the domains that are browsed. I think most DDG users could live with a slightly less then optimal display of favicons in order to gain the assurance, that their domain history is indeed private to their device. That latter is literally the main reason, why most people install your browser in the first place.

I don't think this should be a discussion about "privacy policies". You are integrating a piece of code and infrastructure in your product, that has the capabilities to spy on people. It just shouldn't be there in the first place.

You should not keep this issue just closed like this, this is a serious matter and it is obvious from the community reactions that this needs to be discussed. (https://news.ycombinator.com/item?id=23708166)

[black-snow]

How is this still closed?

[stefanct]

This "feature" has been added in March 2018: https://github.com/duckduckgo/Android/commit/03f99c4a381d880ab850f0ddde9f692057a840e0#diff-63ac5c0d645555fe179e72977d9c1728

And as of now it is still unchanged in the develop branch: https://github.com/duckduckgo/Android/blob/e2f2d54a6b4452277467db403a3546512401b493/app/src/main/java/com/duckduckgo/app/global/UriExtension.kt#L83

(I did not look into the control flow to determine when this gets executed at all though.)

[thors]

I don't know why all the fuss... You all know that if you are using his app, you already "trust" that his search service is respecting your privacy, right? C'mon, they are being transparent, as usual.

1. This is not only about the search service.
2. Trust is only a last resort, after they did everything reasonably possible to avoid acquiring unnecessary data in the first place (which they apparently don't).

There are plenty of good browsers on the market. This browser had once privacy as a main selling point, but as it looks, not any more.

[RokerHRO]

@tagawa : Trust is a very brittle thing. It grows slowly and takes years to built it. And only one line of code can shatter it. But instead to react professionally and contritely you made it worse to stamp on the shards to make sure no useful piece of trust will survive. You had one job! :-(

[bluesign]

Sorry but this is not enough reason. There is a simple question you should ask to yourself.

• Would you be ok to use a third party for this with same privacy policy?

[zink-chimaera]

At DuckDuckGo, we do not collect (...) personal information

Meanwhile DDG literally collecting personal information from my device. 😕

DDG changed security issue: Ignored

[pythoneer]

For those that don't really understand what is happening:

It is really about that code snippet from here:

private const val faviconBaseUrlFormat = "https://icons.duckduckgo.com/ip3/%s.ico"

fun Uri?.faviconLocation(): Uri? {
    val host = this?.host
    if (host.isNullOrBlank()) return null
    return Uri.parse(String.format(faviconBaseUrlFormat, host))
}

The function for retrieving the Favicon is asking for the current host the user is on (e.g. apple.com) and sends it to a service hosted by duckduckgo.com that is saved in the const faviconBaseUrlFormat. %s is replaced with the actual host the user is on and is send to the duckduckgo.com servers. As an example (change it yourself to whatever host you want):

https://icons.duckduckgo.com/ip3/apple.com.ico

I guess they have a cache for already visited hosts etc.

This sends every host the user visits (or the Favicon is requested for) to the duckduckgo.com server. I think it does not really matter what you do or do not do with the data. Its not about trust – its about the minimum you need to trust and this is clearly breaking a lot of peoples perception of what the browser should achieve. As it currently stands, i would consider this even worse of what i imagine googles chrome would do to its users. To make it clear, i don't know exactly what google chrome is doing – its the perception of what i think certain vendors are doing, and i would not expect this from a "privacy first" browser from duckduckgo.com

Tired of being tracked online? We can help.

Please help!

[GarbageHamburger]

99% of the websites I have seen have either

1. a <link rel="shortcut icon">, which directly points to the favicon
2. if not, a /favicon.ico

I really don't see the point of this URL anyhow. It's useful for search results (because you'd have to download every page to parse its favicon otherwise, and your search results are already served directly from DDG so there'd be no point), but using it in tabs is not the right way. I'm willing to say this is just laziness rather than malice, though: rather than writing code to figure out the favicon from the page, they just used whatever they normally used on the search results page.

NOTE: DDG results page uses external-content.duckduckgo.com instead of proxy.duckduckgo.com, and

~$ host proxy.duckduckgo.com
proxy.duckduckgo.com is an alias for external-content.duckduckgo.com.
~$ host icons.duckduckgo.com
icons.duckduckgo.com is an alias for external-content.duckduckgo.com.

They also seem to be caching the most common favicons like https://github.com/duckduckgo/Android/issues/527#issuecomment-652882558 mentioned. i.e. visit "apple" on DDG and the top link has its favicon set as https://duckduckgo.com/assets/icons/favicons/apple.png.

Please reopen and fix.

[lethargosapatheia]

What I find most disturbing is that fact that they haven't said a word on this topic for a year now.

[GarbageHamburger]

What I find most disturbing is that fact that they haven't said a word on this topic for a year now.

I would expect them to respond now that the link blew up on HN.

[alex-tee]

At DuckDuckGo, we do not collect [...] personal information.

first post says otherwise.

[jonathanronen]

We love DDG for privacy, not for favicons. Privacy starts from not collecting data.

[NeverUsedID]

If trust is a balloon, this is a needle found in a haystack...

[tgy]

Adding a comment from Hacker News here that I find relevant when reading this thread.

There's an interesting disease showing up here in the responses.

I accept DDG's statement that this is about a favicon and that they "do not
collect or share any personal information", and despite that, I also agree with
others that DDG should be on the safe side and just stop doing this small
thing. It's just the safer and more moral thing to do (So DDG, as many are
suggesting, plz stop doing it. Today is good).

But... the reaction here is "they made a mistake, let's pile on like kids in a
playground" ignoring the genuinely huger issue of the amount of info and mining
that google et al. do. There's no measure of proportion in the responses,
someone is making a mistake then there's a wolfish, pack-like desire to get 
stuck in and hurt someone.

Which is why politicians rarely admit mistakes, because it's taken as a sign of
weakness, not strength, to admit you were wrong. DDG isn't the big evil on the
web but from reading some of these you'd think it was the 2nd google.

This isn't about DDG, just the proportionality of responses in public errors
and what society you'd like to have.

(no affiliation to DDG)

[b0z1]

I remember when gitlab made a mistake and many commits were lost. It was a huge deal and they made a livestream and fixed it live. Thats transparency!

Thats what i would expect from duckduckgo here. See they made a mistake and fix it.

But closing the issue just like that is a statement. A huge statement. "We dont care about your privacy"

[tagawa]

Sorry for the frustration this has caused. We're re-opening this to update the app to do this locally ASAP. Please see the follow-up comment by our Founder/CEO here: https://news.ycombinator.com/item?id=23711597

[Bonjur]

Thank you, that is the response people here want to hear.

[ganzgustav22]

I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust ist lost. Their CEO can put his statement where the sun doesn't shine.

[GarbageHamburger]

I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust ist lost. Their CEO can put his statement where the sun doesn't shine.

I don't think this kind of cut-throat response is the thing to do. @tagawa already said the issue is being worked on. Of course, it sucks that this was ignored for a year, but fixing it is the right response.

[cassegfault]

https://news.ycombinator.com/item?id=23711597

The CEO of DDG just committed to resolving this by doing favicon lookups locally. This is actually a really solid response from the company. If they follow through, that makes DDG a pretty awesome company for owning up to a mistake and fixing it. Not something you'd see from any of the other major search providers.

[solomoncaygnuyou]

Adding a comment from Hacker News here that I find relevant when reading this thread.

@tgy That comment is a fine example of 'whataboutism'. DuckDuckGo has no control over Google, so it there is no point in rhetorically asking, "Why care about this issue with DuckDuckGo when Google does so much worse?" DuckDuckGo can however improve its own services when there is clearly improvement needed.

Nobody is "piling on" DuckDuckGo for this. You are witnessing a community of privacy-oriented users express dissatisfaction with a subpar response to a clearly contentious implementation in their software. How else do you expect changes to be made to the software? People have to use their voice or else nothing will be done.

As we can see by this thread, our voices were finally heard (a year after the issue was originally opened) when enough people piped up in a small window of time.

[azihassan]

This will exacerbate my anatidaephobia

[RokerHRO]

@tgy :

[…]
But... the reaction here is "they made a mistake, let's pile on like kids in a
playground"  […]
Which is why politicians rarely admit mistakes, […]

As a software developer I know for sure the difference between a "mistake", that happens without intention, and a feature that is always built in with intention.

And, sorry, but you cannot argue that a company that cares about privacy seriously adds an online service that tracks (and can collect) user's behavior only "by accident".

And the answer of the CEO is also not a good excuse for that:

"[…] our services are encrypted and throw away PII like IP addresses by design"

That is just an empty promise that we cannot verify (and you can hear such phrases from nearly every cloud service provider), so it is useless. Or even worse: If the CEO only publish such PR statements, instead of plausible explanations for this issue, it still looks like a feature that was added intentionally and now – oops – someone found it.

[lethargosapatheia]

I would really give them the benefit of the doubt, I wouldn't compare them to google or whatever. Moreover, they would compromise themselves badly, if something really fishy were to be revelead. This is part of the core of their business. I'll put it down to negligence in this case - not that that's an excuse, as I've already said, it's already been a year, yet it's quite different.

[niksmac]

I still don't see a meaningful reason behind this decision, especially from a company which is claiming to be privacy focused. This could've been handled in a better way for sure. Hope they will fix it soon. Nevertheless it was a simple thing which can be avoided altogether.

[seungjulee]

Tritonio opened this issue on 9 Jul 2019

@CDRussell CDRussell closed this on 12 Jul 2019

It took a year, and HN front page to get it resolved.

[u8983478934]

I've always put DDG in the same category of fishy """pro-privacy""" software companies like Brave that their whole raison d'etre is to rely on aggressive marketing attacking big companies like Google.

[thors]

Adding a comment from Hacker News here that I find relevant when reading this thread.

Read it, laughed, and ignored it there (because I didn't want to create an account there just to debunk it)

There's an interesting disease showing up here in the responses.

(So DDG, as many are suggesting, plz stop doing it. Today is good).

Here I agree

But... the reaction here is "they made a mistake, let's pile on like kids in a playground" ignoring the genuinely huger issue of the amount of info and mining that google et al. do.

The main problem is not the mistake they made in implementation. The issue, why people start piling up is that they don't acknowledge it as a mistake. That, imo, justifies to continue screaming until they do change the implementation.

Which is why politicians rarely admit mistakes, because it's taken as a sign of weakness, not strength, to admit you were wrong.

That is bullshit. DDG doesn't receive flack for admitting a mistake, they are exactly receiving flack for not admitting it,

DDG isn't the big evil on the web but from reading some of these you'd think it was the 2nd google.

DDG, like Google, is a company and obliged to maximise its investors profits (well, they are not publicly traded, so maybe not to the same extent). They are not good or evil (neither is Google), they have a selling point (privacy), which they, right now, appear to contradict with their visible actions. If they don't pay attention on their visible actions, in spite of the marketing they do, they might lose their selling point.

This isn't about DDG, just the proportionality of responses in public errors and what society you'd like to have.

No. This is about DDG. They can reopen the issue, change the implementation, and I will immediately stop commenting about this issue (or maybe make positive comments, how they took customer concerns serious)

[sentialx]

doesn't chromium literally have an event for favicon update? WebView even has the favicons database known in Chrome, so it's even possible to get a favicon before a page loads (you can check it by typing chrome://favicon/https://duckduckgo.com) https://developer.android.com/reference/android/webkit/WebViewClient#onPageStarted(android.webkit.WebView,%20java.lang.String,%20android.graphics.Bitmap)

WebChromeClient.onReceivedIcon: https://developer.android.com/reference/android/webkit/WebChromeClient#onReceivedIcon(android.webkit.WebView,%20android.graphics.Bitmap)

[thors]

I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust ist lost. Their CEO can put his statement where the sun doesn't shine.

Good luck finding a better option, privacy-wise...

[Tritonio]

I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust ist lost. Their CEO can put his statement where the sun doesn't shine.

Good luck finding a better option, privacy-wise...

Firefox Focus is IMO better privacy-wise.

[holderbaum]

Thank you, DDG Team, for taking this matter seriously. Very good response from Gabriel on HN! :heart: :partying_face:

[us31t]

[DDG Marketing] We respect your privacy, we are a privacy company! We fight BigCorp with transparency! [DDG Dev] Ok, lets build a feature! Hmm... what could we do next? Ahhh I got a great idea! Lets send the entire users browser history to our servers, so we can ... hmmm lets see ... how could we argue this ... a yes! Show a favicon! [OS Community] Files Ticket: Hey guys you are sending the entire browser history to your servers, is that by mistake? Could you please anyway stop that? You said you respect our privacy! [DDG Dev] Who are you? What are you talking about? There is nothing wrong with us sending your browser history to our server! We have a privacy policy, can`t you read? This is not a bug this is a feature! - Ticket Close [OSS Community] WFT just happened? Are you nuts? [One year Later] Topic shows up on HN [DDG Management] Oh hello I just woke up (after one year?), I am new to this issue and I do not know what is going on, because I am the CEO! We will remove the feature again! We really did not intend to develop a feature! We are new in this whole privacy thing! Please forgive us. Instead you really should trust us again! See we have this [Enter_A_Super_Duper_Secret_Privacy_Buzzword_That_Is_Not_Available_For_Verification] process, that removes ALL personal information! Really! I am the CEO! CEOs are always right and never lie in public! Could you please stop flaming now? [Some Trolls] Thank you DDG, that you are talking this seriously! [OSS Community] WFT just happened? Are you nuts?

[TBC] - This August on Netflix!