agent模式无法播放youtube的视频

koolwiki commented 5 years ago

如题，测试了一下agent模式，感觉挺好的。有个问题就是打开youtube网页后，随便点开一个视频都无法播放视频，一直处于转圈状态，网页上其他文字和图片都不受影响，只是没法播放视频。信息如下：

D:\goflyway>goflyway.exe -up=agent://testdomain.com:443 -lv dbg0 2018-10-23 18:43:09.126 CST,main.go:267, INFO ,goflyway 181021015134 2018-10-23 18:43:09.151 CST,main.go:270, WARNING ,Please change the default password: -k= 2018-10-23 18:43:09.192 CST,main.go:280, DEBUG ,"ACL chinalist.txt: 0 black rules, 40925 white rules, 0 gray rules" 2018-10-23 18:43:09.193 CST,main.go:436, INFO ,Upstream config: agent://testdomain.com:443 2018-10-23 18:43:09.194 CST,main.go:475, WARNING ,"Agent is not safe, don't use it to access your important data" 2018-10-23 18:43:09.194 CST,main.go:416, INFO ,Client 72727e8 started: you->mitm->0.0.0.0:8100->testdomain.com:443(agent) 2018-10-23 18:43:09.194 CST,main.go:412, INFO ,Web console started at 127.0.0.1:8110 2018-10-23 18:43:19.769 CST,mitm.go:50, DEBUG ,Self signing: www.youtube.com 2018-10-23 18:43:19.786 CST,mitm.go:375, DEBUG ,Agent - GET ssl://www.youtube.com:443 2018-10-23 18:43:20.151 CST,mitm.go:375, DEBUG ,Agent - GET ssl://www.youtube.com:443 2018-10-23 18:43:20.192 CST,mitm.go:50, DEBUG ,Self signing: i.ytimg.com 2018-10-23 18:43:20.210 CST,mitm.go:375, DEBUG ,Agent - GET ssl://i.ytimg.com:443 2018-10-23 18:43:20.741 CST,mitm.go:50, DEBUG ,Self signing: clients4.google.com 2018-10-23 18:43:20.773 CST,mitm.go:375, DEBUG ,Agent - POST ssl://clients4.google.com:443 2018-10-23 18:43:21.319 CST,mitm.go:336, ERROR ,"IO copy 0 bytes: write 127.0.0.1:3172, Software caused connection abort" 2018-10-23 18:43:21.545 CST,mitm.go:50, DEBUG ,Self signing: raw.githubusercontent.com 2018-10-23 18:43:21.564 CST,mitm.go:375, DEBUG ,Agent - GET ssl://raw.githubusercontent.com:443 2018-10-23 18:43:21.610 CST,mitm.go:375, DEBUG ,Agent - POST

coyove commented 5 years ago

这个bug应该是已经修掉过的，至少我这里油管正常。你可以打开一个视频，再打开开发者工具，观察一下XHR请求的结果是什么

koolwiki commented 5 years ago

显示Failed to load response data。

POST https://www.youtube.com/api/stats/qoe?event=streamingstats&fmt=278&afmt=250&cpn=FKsyG-aP2qNUY1zG&ei=YwbPW7DNN7K_j-8PoYSn-AQ&el=detailpage&docid=ogCe6sC6_cA&ns=yt&fexp=23705256%2C23710476%2C23717918%2C23721698%2C23721898%2C23733015%2C23744176%2C23744530%2C23751767%2C23752869%2C23755886%2C23755898%2C23758138%2C23760521%2C23762649%2C23762813%2C23763106%2C23763126%2C23763691%2C23764135%2C23764993%2C23767581%2C23768218%2C23768325%2C23770185%2C23770310%2C23771604%2C23772115%2C23772221%2C23772350%2C23772831%2C23773088%2C3300108%2C3300131%2C3300161%2C3313321%2C3314088%2C9406006%2C9407156%2C9407610%2C9449243%2C9466591%2C9469036%2C9471239%2C9485000&cl=217969278&seq=21&c=WEB&cver=2.20181020&cplayer=UNIPLAYER&cbr=Chrome&cbrver=70.0.3538.67&cos=Windows&cosver=10.0&bwe=208.949:6185,210.001:6185&bat=208.949:0.72:0,210.001:0.72:0&cmt=208.949:0.000,210.001:0.000&bh=208.949:0.000,210.001:0.000 net::ERR_BLOCKED_BY_CLIENT

coyove commented 5 years ago

使用了2.0.0rc1的包依旧没有重现问题。请尝试一下更换php hosting看看

koolwiki commented 5 years ago

用的也是000webhostapp.com，我再看看吧，谢谢！打开127.0.0.1:8110看代理规则发现有些googlevideo.com的域名无法解析出ip：

coyove commented 5 years ago

控制台里面显示的全是unknown吗？即便是可以访问的网站也是unknown？

koolwiki commented 5 years ago

是的，能上的网站也是显示Unknow，比如google。换了电脑也不行，使用000webhostapp.com同一个host下其他的php代理脚本可以看油管的视频。

coyove commented 5 years ago

我想到两点：

通过最新的源码运行，观察是否出现同样情况
通过wireshark或类似的软件抓本地客户端127.0.0.1:8100发出的包，这些包是响应浏览器的请求的，其数据里面HTTP头的connection值应该总为close。之前修这个bug时，正是因为youtube返回的是keep-alive，才导致视频片段无法缓冲的问题。(wireshark mitmproxy需要加载ca.pem）

koolwiki commented 5 years ago

最新的源代码是指什么？我目前下载了release里面最新的RC版，自己也编译了一下，但都出现同样的问题。
我后面测试一下再来反馈。

非常感谢！

coyove commented 5 years ago

如果是最新的代码，请观察一下web控制台里还是unknown吗，同时你可以把index.php的内容改为：

<?php
echo gethostbyname("googlevideo.com");
?>

然后访问它，看看能不能获得IP。

另外在代码里：https://github.com/coyove/goflyway/blob/master/proxy/mitm.go#L324 后加上一行

proxy.Logger.Dbg0(string(buf))

然后使用-lv dbg0^dbg^info^log启动goflyway，观察输出的结果

koolwiki commented 5 years ago

能够获取ip。
加上代码后可以编译成功，但是使用-lv dbg0^dbg^info^log运行后自动退出（不使用可以正常运行并能够打开google等网站，但还是看不了油管视频），信息如下： D:\goflyway>goflyway_windows_debug.exe -lv dbg0^dbg^info^log panic: unexpected log level: dbg0dbginfolog

goroutine 1 [running]: github.com/coyove/common/logg.(*Logger).SetLevel(0x10fd6210, 0x10fd4090, 0xe, 0x1) /home/xwfunny/work/src/github.com/coyove/common/logg/logg.go:71 +0x27a github.com/coyove/common/logg.NewLogger(0x10fd4090, 0xe, 0x20) /home/xwfunny/work/src/github.com/coyove/common/logg/logg.go:110 +0x1b1 main.main() /home/xwfunny/work/src/github.com/coyove/goflyway/cmd/goflyway/main.go:255 +0xd1

coyove commented 5 years ago

-lv 后面的参数需要用双引号括起来

koolwiki commented 5 years ago

输出的信息如下： D:\goflyway>goflyway_windows_debug.exe -up=agent://xxxxx.com:443 -lv "dbg0^dbg^info^log" 2018-10-24 20:19:19.113 CST,main.go:270, WARNING ,Please change the default password: -k= 2018-10-24 20:19:19.208 CST,main.go:475, WARNING ,"Agent is not safe, don't use it to access your important data" 2018-10-24 20:19:23.058 CST,mitm.go:325, DEBUG0 ,"HTTP/1.1 404 Not Found Content-Length: 1589 Alt-Svc: quic="":443""; ma=2592000; v=""44,43,39,35"" Connection: close Content-Type: text/html; charset=UTF-8 Date: Wed, 24 Oct 2018 12:19:18 GMT Referrer-Policy: no-referrer

" 2018-10-24 20:19:23.439 CST,mitm.go:325, DEBUG0 ,"HTTP/1.1 200 OK Content-Length: 135082 Accept-Ranges: bytes Alt-Svc: quic="":443""; ma=2592000; v=""44,43,39,35"" Cache-Control: public, max-age=21600 Connection: close Content-Type: image/webp Date: Wed, 24 Oct 2018 12:19:18 GMT Etag: ""1539180464"" Expires: Wed, 24 Oct 2018 18:19:18 GMT Server: sffe X-Content-Type-Options: nosniff X-Xss-Protection: 1; mode=block

" 2018-10-24 20:19:25.896 CST,mitm.go:309, ERROR ,"Round trip www.youtube.com: read 104.18.32.121:443, Connection reset by peer" 2018-10-24 20:19:27.024 CST,mitm.go:325, DEBUG0 ,"HTTP/1.1 200 OK Transfer-Encoding: chunked Alt-Svc: quic="":443""; ma=2592000; v=""44,43,39,35"" Cache-Control: no-cache Connection: close Content-Encoding: br Content-Security-Policy-Report-Only: frame-ancestors https://*.youtube.com; report-uri https://www.youtube.com/csp_204?t=framing Content-Type: text/html; charset=utf-8 Date: Wed, 24 Oct 2018 12:19:21 GMT Expires: Tue, 27 Apr 1971 19:44:06 EST Server: YouTube Frontend Proxy Strict-Transport-Security: max-age=31536000 X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-Xss-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube

" 2018-10-24 20:19:28.874 CST,mitm.go:325, DEBUG0 ,"HTTP/1.1 200 OK Content-Length: 1552 Access-Control-Allow-Credentials: true Access-Control-Allow-Origin: https://www.youtube.com Alt-Svc: quic="":443""; ma=2592000; v=""44,43,39,35"" Cache-Control: no-cache Connection: close Content-Encoding: br Content-Security-Policy-Report-Only: frame-ancestors https://*.youtube.com; report-uri https://www.youtube.com/csp_204?t=framing Content-Type: text/plain; charset=UTF-8 Date: Wed, 24 Oct 2018 12:19:23 GMT Expires: Tue, 27 Apr 1971 19:44:06 EST Server: YouTube Frontend Proxy Strict-Transport-Security: max-age=31536000 X-Content-Type-Options: nosniff

coyove commented 5 years ago

你用chrome的隐身窗口试一下，看看结果是什么

koolwiki commented 5 years ago

隐身模式也打不开，我手机上用arm版测试也无法打开油管视频。我怀疑是我的php设置有问题，我能把我的host网址发给你测试一下吗？

coyove commented 5 years ago

从dbg0输出的结果来看并不是host的问题，不过你可以给我测试一下：coyove@hotmail.com

coyove commented 5 years ago

今天更新了一下chrome，终于出现了同样的问题，正在排查中。你可以试试http模式，看看能不能正常工作

koolwiki commented 5 years ago

http，https和cf模式一直都工作得很好。期待大佬早日找出bug！

ps：奇怪的是我用edge，ie等浏览器也会出现不能播放的问题。

coyove commented 5 years ago

是的，IE，firefox，safari都不行，应该是youtube返回了什么特殊的http头阻止了请求而不是浏览器的问题。访问vimeo之类的视频网站是可以的。

coyove / goflyway

agent模式无法播放youtube的视频 #115