search

TeamNewPipe / NewPipe

A libre lightweight streaming front-end for Android.
https://newpipe.net
GNU General Public License v3.0
31.51k stars 3.06k forks source link

IP address leak! (Tor) (reCAPTCHA) #3612

Open ghost opened 4 years ago

ghost commented 4 years ago

Version

Steps to reproduce the bug

  1. Use NetGuard to forward everything through Tor. In this case NewPipe
  2. Open a video
  3. Get reCAPTCHA challenge
  4. Tor IP
  5. Complete the challenge
  6. (Realize the reCAPTCHA is broken :/)
  7. Real IP

Expected behavior

Pretty harsh, but my expectation would be that NewPipe wouldn't leak my real IP to Google at ANY cost. Ya, I made a video recording this \o/

Actual behaviour

Would like my IP hidden from Google. Please.

Screenshots/Screen recordings

Same video, different hosts https://anonfile.com/B9sfZezdo0/oh_no_webm https://streamable.com/p6h6we

Logs

Stypox commented 4 years ago

I don't get how this could be possible, since NewPipe never sends any ip explicitly. Could you capture a logcat of this happening with a debug apk (e.g. this one from #3478)?

ghost commented 4 years ago

Of course :) I'll get right onto it.

ghost commented 4 years ago

adb logcat > llogcatnewpipe.txt llogcatnewpipe.txt

Please tell me if I did something wrong or if I should add any command options

Stypox commented 4 years ago

Here I extracted the recaptcha-related data:

05-18 11:57:16.976   747  1088 I ActivityManager:                            START u0 {cmp=org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity (has extras)} from uid 10145
05-18 11:57:31.663   747  1682 I ActivityManager:                            START u0 {cmp=org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity (has extras)} from uid 10145
05-18 12:00:47.660  3245  3245 D DefaultKioskFragment@a78ef85:               onError() called with: exception = [org.schabi.newpipe.extractor.exceptions.ReCaptchaException: reCaptcha Challenge requested]
05-18 12:00:47.661  3245  3245 D DefaultKioskFragment@a78ef85:               onReCaptchaException() called
05-18 12:00:47.678   747  1310 I ActivityManager:                            START u0 {cmp=org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity (has extras)} from uid 10145

05-18 12:00:49.234  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: shouldOverrideUrlLoading: request.url=https://www.google.com/sorry/index?continue=https://www.youtube.com/results%3Fsearch_query%3Dtest%26pbj%3D1&q=EgS53GWLGJC6ifYFIhkA8aeDSyJiiE001w2n9x67c5NPMC3rQ6vVMgFy
05-18 12:00:49.234  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index?continue=https://www.youtube.com/results%3Fsearch_query%3Dtest%26pbj%3D1&q=EgS53GWLGJC6ifYFIhkA8aeDSyJiiE001w2n9x67c5NPMC3rQ6vVMgFy
05-18 12:00:49.238  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=null
05-18 12:00:51.764  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index?continue=https://www.youtube.com/results%3Fsearch_query%3Dtest%26pbj%3D1&q=EgS53GWLGJC6ifYFIhkA8aeDSyJiiE001w2n9x67c5NPMC3rQ6vVMgFy
05-18 12:00:51.766  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=null
05-18 12:00:58.702  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index
05-18 12:00:58.706  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=NID=204=kRglBs369QjPd51u392P51RN71W2hprYkRnQrMYo300luUKC_aWKew1tye9_7VwThXL9B3AJBpYdQ1gC9nd8mg3s3iLbg1hf-NsEkgxPBW3dNWoErb9G7bmEFFiBUiqq5Kw9OWkCLyBiFwKsUEdhhypkgNLS9700xZBBtnjeiDI
05-18 12:01:23.415  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index
05-18 12:01:23.416  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=NID=204=kRglBs369QjPd51u392P51RN71W2hprYkRnQrMYo300luUKC_aWKew1tye9_7VwThXL9B3AJBpYdQ1gC9nd8mg3s3iLbg1hf-NsEkgxPBW3dNWoErb9G7bmEFFiBUiqq5Kw9OWkCLyBiFwKsUEdhhypkgNLS9700xZBBtnjeiDI
05-18 12:01:23.416  3245  3245 D class org.schabi.newpipe.ReCaptchaActivity: saveCookiesAndFinish:     foundCookies=

05-18 12:01:47.391  3380  3380 D DefaultKioskFragment@f2a3cfc:               onError() called with: exception = [org.schabi.newpipe.extractor.exceptions.ReCaptchaException: reCaptcha Challenge requested]
05-18 12:01:47.392  3380  3380 D DefaultKioskFragment@f2a3cfc:               onReCaptchaException() called
05-18 12:01:47.409   747  1683 I ActivityManager:                            START u0 {cmp=org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity (has extras)} from uid 10145

05-18 12:01:49.033  3380  3380 D class org.schabi.newpipe.ReCaptchaActivity: shouldOverrideUrlLoading: request.url=https://www.google.com/sorry/index?continue=https://www.youtube.com/results%3Fsearch_query%3Dtest%26pbj%3D1&q=EgS53GWLGMy6ifYFIhkA8aeDS9F75V0Stol0xr5A8qoJLtgI5LIJMgFy
05-18 12:01:49.033  3380  3380 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index?continue=https://www.youtube.com/results%3Fsearch_query%3Dtest%26pbj%3D1&q=EgS53GWLGMy6ifYFIhkA8aeDS9F75V0Stol0xr5A8qoJLtgI5LIJMgFy
05-18 12:01:49.037  3380  3380 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=null
05-18 12:01:51.378  3380  3380 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index?continue=https://www.youtube.com/results%3Fsearch_query%3Dtest%26pbj%3D1&q=EgS53GWLGMy6ifYFIhkA8aeDS9F75V0Stol0xr5A8qoJLtgI5LIJMgFy
05-18 12:01:51.381  3380  3380 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=null
05-18 12:01:58.363  3380  3380 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index
05-18 12:01:58.366  3380  3380 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=NID=204=fx4snLbUHmXBxA4_nukjuUMkEhcKqKekAiYY1i5p5IGE95JGb5j08Qw4Xzn98AluWlN5O0Z4fmrZsI1u2tBhWhCyc6Arl7XAy5WyjMlSvSNzaB1w1YxiWBeuv6RDNx-Cx1o2zm0MbmE_65HgG3ayo4usvEjkuEehC12g9XXLVs4

05-18 12:02:53.014   435  1187 W SurfaceFlinger:                             Attempting to set client state on removed layer: org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity#0
05-18 12:02:53.014   435  1187 W SurfaceFlinger:                             Attempting to destroy on removed layer: org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity#0
05-18 12:02:53.017   435   865 W SurfaceFlinger:                             Attempting to set client state on removed layer: org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity#0
05-18 12:02:53.018   435   865 W SurfaceFlinger:                             Attempting to destroy on removed layer: org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity#0
05-18 12:02:53.379   747  1682 I WindowManager:                              WIN DEATH: Window{a45adef u0 org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity}
05-18 12:02:53.391   435  1546 W SurfaceFlinger:                             Attempting to destroy on removed layer: AppWindowToken{9bc627a token=Token{85a7ea5 ActivityRecord{5aa149c u0 org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity t1616}}}#0
05-18 12:02:59.057  3492  3492 D DefaultKioskFragment@f2a3cfc:               onError() called with: exception = [org.schabi.newpipe.extractor.exceptions.ReCaptchaException: reCaptcha Challenge requested]
05-18 12:02:59.058  3492  3492 D DefaultKioskFragment@f2a3cfc:               onReCaptchaException() called
05-18 12:02:59.074   747   758 I ActivityManager:                            START u0 {cmp=org.schabi.newpipe.debug.release/org.schabi.newpipe.ReCaptchaActivity (has extras)} from uid 10145

05-18 12:03:00.453  3492  3492 D class org.schabi.newpipe.ReCaptchaActivity: shouldOverrideUrlLoading: request.url=https://www.google.com/sorry/index?continue=https://www.youtube.com/results%3Fsearch_query%3Dtest%26pbj%3D1&q=EgS53GWLGJS7ifYFIhkA8aeDS-l79vt5A-rNVGKn2SGL8NSJ6K14MgFy
05-18 12:03:00.454  3492  3492 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index?continue=https://www.youtube.com/results%3Fsearch_query%3Dtest%26pbj%3D1&q=EgS53GWLGJS7ifYFIhkA8aeDS-l79vt5A-rNVGKn2SGL8NSJ6K14MgFy
05-18 12:03:00.458  3492  3492 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=null
05-18 12:03:02.970  3492  3492 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index?continue=https://www.youtube.com/results%3Fsearch_query%3Dtest%26pbj%3D1&q=EgS53GWLGJS7ifYFIhkA8aeDS-l79vt5A-rNVGKn2SGL8NSJ6K14MgFy
05-18 12:03:02.972  3492  3492 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=null
05-18 12:03:08.104  3492  3492 D class org.schabi.newpipe.ReCaptchaActivity: handleCookiesFromUrl:     url=https://www.google.com/sorry/index
05-18 12:03:08.106  3492  3492 D class org.schabi.newpipe.ReCaptchaActivity: handleCookies:            cookies=NID=204=iMEqZ2hPp5ik2u8a8gylclkWq_ZqanZbe8HdTXNOG4rxzzpPxguj4pnyfBMXYCNGOVgMit7rSoOGhXdRTHRj59Lpqd0CxjQBcjz8p_Qp-gqWe8183SNUQ9ZavE1ptWODeh6gTK7po7-mmov7EzD8Cbo2Wrij8XN49KRq79rBZjk
Stypox commented 4 years ago

This is really strange: there is no data at all showing that an ip has been saved in cookies or something similar, so that can't be the cause (you can see saveCookiesAndFinish: foundCookies= nothing). So this could have something to do with the recaptcha javascript code running in the webview. I currently have no idea how this problem could be fixed.

ghost commented 4 years ago

Oh that's weird.

As you said, it's probably the JavaScript causing this. And disabling JavaScript on reCAPTCHA page would make the page nonfunctional. I guess this is one of those issues we can't fix then huh

ghost commented 4 years ago

Unless WebView uses WebRTC which is known for leaking IP addresses. This Reddit link is having the same problem.

Any Android app that uses WebView should be affected by WebRTC IP leak

Some people may go to the extent to disable Android System WebView (may require root), this will cripple usability.

TheAssassin commented 4 years ago

@ingingin do you have any actual proof that could help identify the issue, if there is any? How did you notice your IP was leaked? I don't see any such evidence in the original issue description.

Generally, we take privacy serious and try to avoid data leakage, but please don't expect full protection. There is always a risk for a leak. If in doubt, you need to use different tools on an actually hardened system, e.g., the Tails linux distro on a safe computer. Android devices, generally anything mobile, is not 100% safe anyway.

ghost commented 4 years ago

What do you mean, actual proof? I noticed my IP was leaked because as I mentioned in the video, before doing all that, I had the IP address which Tor gave me, then suddently I get my own IP printed. No I don't have evidence to backup my statement cuz I don't wanna show my IP to anyone.

But using NetGuard, blocking access to every system app. On a non-googled tablet, running LineageOS. And only allowing NewPipe Internet access while routing through Tor on 127.0.0.1 port 9050. Tested and working.

Should we then give the android VPN implementation shit for this? I know these problems about the risks on tablets and phones generally but this feels like something more complicated. While I don't understand any of androids interior I would trust you know much more, which is why I really wouldn't know why or HOW its happening :/

ghost commented 4 years ago

But I know that Google displays your IP when getting reCAPTCHAs on the webpage too. The same thing happens on NewPipe. And I would guess you guys use WebView? Something about that messes up the, something. Now instead of using YouTube, wouldn't it be better to add Invidious? As far I remember you don't get any reCAPTCHAs from that. Which would solve this ?_?

ale5000-git commented 4 years ago

I'm not sure but I think the problem is NetGuard, if the requests done in the WebView are forwarded through Tor it shouldn't see your IP.

Also have you tried to install a different WebView and switch the default WebView to the new one?

ale5000-git commented 4 years ago

On a real browser the uBlock Origin add-on can prevent the IP leak, is it possible to replicate the behaviour?

See here: https://github.com/gorhill/uBlock/wiki/Prevent-WebRTC-from-leaking-local-IP-address

ghost commented 4 years ago

@ale5000-git Yeah, you just got to thinking again, I believe I didn't route the system through Tor actually, but using WebRTC with Tor, should leak my IP anyhow, because of the VPN situation ?? I can't think of a way to replicate that thing uBlock Origin uses on Android. Haven't seen anything like that.

Right now I'm trying to see if using Bromite's WebView would make a difference. I believe it's because I didn't actually route Android's WebView through Tor which I actually thought I did. And only having NewPipe through Tor and then WebView not, would mess up things.

Coming back tomorrow with news

ghost commented 4 years ago

I'm an idiot...I think

ghost commented 4 years ago

Okay I'm back, this is not tomorrow. But yes, even with Android System WebView denied Internet access + the whole system. It still leaks my IP address.

I also tried the same with Bromite's WebView from my Android phone. And it didn't leak my IP. Or so I don't think. The real problem goes down to the reCAPTCHA being broken on my Tablet. Because on my phone it works, the ✔️ mark is displayed. And the page doesnt refresh. On my Tablet, the reCAPTCHA box disappears completely and refreshes the page kinda. But I'll try to install Bromite's WebView onto my Tablet and see if that makes the difference. But if that's the cause. This issue can't be resolved right?

ale5000-git commented 4 years ago

In my opinion the safest option would be to include a web rendering engine directly inside NewPipe code but that would certainly add a lot of complications.

ghost commented 4 years ago

Wouldn't Invidious implementation solve this? I would vouch for that instead of a real fix for this. As Invidious doesn't seem to require any reCAPTCHAs because you're not accessing the Google captcha page. Going directly to googlevideo is great. And having Tor with that 👍

Or, I don't understand this, so I wouldn't know.