search

XX-net / XX-Net

A proxy tool to bypass GFW.
33k stars 7.7k forks source link

使用多线程分片下载文件时会出现问题 #3151

Open 821938089 opened 8 years ago

821938089 commented 8 years ago

如果用多线程文件经常只能下到开头的一点点，后面完全下不了，但是看加速球是正在下载的。 但是如果是用单线程又正常了，文件一直都正常下载，但是速度很慢。 希望能解决这个蛋疼的问题。 不知道是不是ip的问题，但是我看了h1连接池有20几个ip，h2有2个ip，ip这么多，应该是没问题的。

XX-Net Status:

sys-platform: x86, Windows-7-6.1.7601-SP1 os-system: Windows os-version: 6.1.7601 os-release: 7 os-detail: Version:6-1; Build:7601; Platform:2; CSD:Service Pack 1; ServicePack:1-0; Suite:256; ProductType:0 architecture: 32bit,WindowsPE browser: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 xxnet-version: 3.1.0

python-version: 2.7.11 openssl-version: 0.15.1 TLSv1_2 h2:alpn

ipv6-status: 0 gws-ip-num: total:1439 good:460 network-status: OK connected-link: new:0 used:0 worker: h1:22 h2:2 scan-ip-thread-num: 50 ip-quality: 618 is-idle: 0 block-stat: OK proxy_state: OK ca_state: OK Appid_Working: true Appids_Out_Of_Quota: false Appids_Not_Exist: false Using_Public_Appid: false

xxnet commented 8 years ago

​你怎么切换到单线程的？

日志有没有异常信息？

jzp820927 commented 8 years ago

我估计他是不是用的下载工具，在下载工具里面设置的下载线程数，比如 IDM 之类的

821938089 commented 8 years ago

@xxnet manual.ini文件[autorange]里的threads调成1就是单线程 从日志里挑出来的异常信息: May 02 16:51:51.782 - [ERROR] ssl send:Error([('SSL routines', 'ssl3_write_bytes ', 'ssl handshake failure')],) May 02 16:51:51.784 - [ERROR] Except stack:Traceback (most recent call last): File "F:\XX-Net-3.1.0\code\default\gae_proxy\local\openssl_wrap.py", line 95, in send return self.iowait(self._connection.send, data, flags) File "F:\XX-Net-3.1.0\code\default\gae_proxy\local\openssl_wrap.py", line 80, in __iowait raise e Error: [('SSL routines', 'ssl3_write_bytes', 'ssl handshake failure')]

May 02 16:55:09.595 - [ERROR] 208.117.228.105 head b44564808 request fail:Except ion('app check fail 502',) May 02 16:55:09.662 - [ERROR] Except stack:Traceback (most recent call last): File "F:\XX-Net-3.1.0\code\default\gae_proxy\local\http1.py", line 158, in hea d_request raise Exception("app check fail %r" % status) Exception: app check fail 502

May 02 16:56:52.632 - [WARNING] 217.18.145.114 _request bad status line:BadStatu sLine("''",)

May 02 16:57:16.860 - [WARNING] fetch gae status:400 url:https://r1---sn-vgqsen7 s.googlevideo.com/videoplayback?id=o-AOiNN-GKnaxtwlYZ5rm7QTWUzaOURJkmCgpdQY-hMoD v&requiressl=yes&pl=27&ratebypass=yes&ip=107.178.194.110&source=youtube&upn=vtog aPNMlto&itag=22&mime=video/mp4&dur=1915.344&fexp=9416126,9416891,9422596,9428398 ,9431012,9433096,9433946,9434290&sver=3&signature=6B8FC8BE61B015724FD1D8D5A3AB5A 428F459B28.2E137558F08E12A89F26B68BAB76560250A236BA&ipbits=0&sparams=dur,expire, id,initcwndbps,ip,ipbits,ipbypass,itag,lmt,mime,mm,mn,ms,mv,nh,pl,ratebypass,req uiressl,source,upn&expire=1462191132&lmt=1462118851198277&key=cms1&title=Awesome +Gold%2FGhast+Farm+(up+to+2%2C000+tears%2F250+gold+blocks+per+hour)&redirect_cou nter=1&req_id=fe18baf525c5a3ee&cms_redirect=yes&ipbypass=yes&mm=31&mn=sn-vgqsen7 s&ms=au&mt=1462169545&mv=m May 02 16:57:16.862 - [DEBUG] 203.233.96.55 Send:DataFrame 25 F:END_STREAM len:8 53 May 02 16:57:16.862 - [DEBUG] 203.233.96.55 Recv:PingFrame May 02 16:57:16.866 - [WARNING] IP:203.233.96.55 not support GAE, server type: May 02 16:57:16.868 - [DEBUG] 203.233.96.55 Send:HeadersFrame 27 F:END_HEADERS May 02 16:57:16.868 - [DEBUG] 203.233.96.55 Recv:HeadersFrame 13 F:END_HEADERS May 02 16:57:16.870 - [INFO] remove ip:203.233.96.55 left amount:1670 gws_num:16 70 May 02 16:57:16.872 - [DEBUG] 203.233.96.55 Send:DataFrame 27 F:END_STREAM len:8 51 May 02 16:57:16.874 - [DEBUG] 203.233.96.55 Recv:DataFrame 13 len:15919 May 02 16:57:16.877 - [DEBUG] 203.233.96.55 worker close:ip not support GAE May 02 16:57:16.879 - [WARNING] RangeFetch bytes=131073-262144 return False May 02 16:58:15.375 - [ERROR] 202.75.147.42 head pnj201416301 request fail:SysCa llError(10053, 'WSAECONNABORTED') May 02 16:58:15.416 - [ERROR] Except stack:Traceback (most recent call last): File "F:\XX-Net-3.1.0\code\default\gae_proxy\local\http1.py", line 153, in hea d_request response.begin() File "F:\XX-Net-3.1.0\code\default\python27\1.0\lib\httplib.py", line 453, in begin version, status, reason = self._read_status() File "F:\XX-Net-3.1.0\code\default\python27\1.0\lib\httplib.py", line 409, in _read_status line = self.fp.readline(_MAXLINE + 1) File "F:\XX-Net-3.1.0\code\default\python27\1.0\lib\socket.py", line 480, in r eadline data = self._sock.recv(self._rbufsize) File "F:\XX-Net-3.1.0\code\default\gae_proxy\local\openssl_wrap.py", line 118, in recv return self.__iowait(self._connection.recv, bufsiz, flags) File "F:\XX-Net-3.1.0\code\default\gae_proxy\local\openssl_wrap.py", line 80, in __iowait raise e SysCallError: (10053, 'WSAECONNABORTED') May 02 16:59:15.313 - [WARNING] parse request fail:)?

May 02 16:59:15.315 - [WARNING] unhandler cmd:None path:mtalk.google.com:5228 fr om:127.0.0.1:3667 May 02 17:02:33.813 - [WARNING] 113.171.240.91 read timeout t:344402 len:131454 left:123352 然后就下不了了，h2连接池仅有的2个ip变成0了，h1连接池还有很多ip

821938089 commented 8 years ago

@jzp820927 下载工具是谷歌浏览器自带的下载

xxnet commented 8 years ago

​试试 3.1.1

821938089 commented 8 years ago

不对，我刚刚搞错了点东西..... 日志是有的

821938089 commented 8 years ago

@xxnet 3.1.1依旧不行 日志:May 02 17:53:19.719 - [INFO] >>>>>>>>>>>>>>> RangeFetch started('https://r1---sn-vgqsen7s.googlevideo.com/videoplayback?id=o-AOiNN-GKnaxtwlYZ5rm7QTWUzaOURJkmCgpdQY-hMoDv&requiressl=yes&pl=27&ratebypass=yes&ip=107.178.194.110&source=youtube&upn=vtogaPNMlto&itag=22&mime=video/mp4&dur=1915.344&fexp=9416126,9416891,9422596,9428398,9431012,9433096,9433946,9434290&sver=3&signature=6B8FC8BE61B015724FD1D8D5A3AB5A428F459B28.2E137558F08E12A89F26B68BAB76560250A236BA&ipbits=0&sparams=dur,expire,id,initcwndbps,ip,ipbits,ipbypass,itag,lmt,mime,mm,mn,ms,mv,nh,pl,ratebypass,requiressl,source,upn&expire=1462191132&lmt=1462118851198277&key=cms1&title=Awesome+Gold%2FGhast+Farm+(up+to+2%2C000+tears%2F250+gold+blocks+per+hour)&redirect_counter=1&req_id=fe18baf525c5a3ee&cms_redirect=yes&ipbypass=yes&mm=31&mn=sn-vgqsen7s&ms=au&mt=1462169545&mv=m') 0-1048576 May 02 17:53:19.792 - [INFO] >>>>>>>>>>>>>>> [thread 3296] 1048577 bytes 0-1048576/391980119 May 02 17:53:19.794 - [INFO] >>>>>>>>>>>>>>> Successfully reached 1048576 bytes. May 02 17:53:20.155 - [DEBUG] 210.61.221.182 Send:HeadersFrame 5 F:END_HEADERS May 02 17:53:20.157 - [DEBUG] 210.61.221.182 Send:DataFrame 5 F:END_STREAM len:852 May 02 17:53:20.634 - [DEBUG] 210.61.221.182 Recv:HeadersFrame 5 F:END_HEADERS May 02 17:53:20.644 - [DEBUG] 210.61.221.182 Send:HeadersFrame 7 F:END_HEADERS May 02 17:53:20.646 - [DEBUG] 210.61.221.182 Send:DataFrame 7 F:END_STREAM len:852 May 02 17:53:20.860 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:15260 May 02 17:53:21.155 - [DEBUG] 210.61.221.182 Send:HeadersFrame 9 F:END_HEADERS May 02 17:53:21.157 - [DEBUG] 210.61.221.182 Send:DataFrame 9 F:END_STREAM len:853 May 02 17:53:21.187 - [DEBUG] ip:110.164.13.102 real fail May 02 17:53:21.446 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:21.555 - [DEBUG] 210.61.221.182 Send:HeadersFrame 11 F:END_HEADERS May 02 17:53:21.557 - [DEBUG] 210.61.221.182 Send:DataFrame 11 F:END_STREAM len:854 May 02 17:53:21.842 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:23.242 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:23.243 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:640 May 02 17:53:23.930 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:15744 May 02 17:53:25.012 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:26.351 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16370 May 02 17:53:27.387 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:15374 May 02 17:53:28.379 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:29.756 - [DEBUG] restore ip:113.171.241.83 May 02 17:53:29.962 - [INFO] GAE CONNECT ssl.gstatic.com:443 May 02 17:53:29.984 - [DEBUG] GAE CONNECT GET https://ssl.gstatic.com/docs/common/cleardot.gif?zx=72j7jo185dau May 02 17:53:29.987 - [DEBUG] 210.61.221.182 Send:HeadersFrame 13 F:END_HEADERS May 02 17:53:29.989 - [DEBUG] 210.61.221.182 Send:DataFrame 13 F:END_STREAM len:352 May 02 17:53:30.664 - [INFO] GAE CONNECT www.twitter.com:443 May 02 17:53:30.789 - [WARNING] RangeFetch "GET https://r1---sn-vgqsen7s.googlevideo.com/videoplayback?id=o-AOiNN-GKnaxtwlYZ5rm7QTWUzaOURJkmCgpdQY-hMoDv&requiressl=yes&pl=27&ratebypass=yes&ip=107.178.194.110&source=youtube&upn=vtogaPNMlto&itag=22&mime=video/mp4&dur=1915.344&fexp=9416126,9416891,9422596,9428398,9431012,9433096,9433946,9434290&sver=3&signature=6B8FC8BE61B015724FD1D8D5A3AB5A428F459B28.2E137558F08E12A89F26B68BAB76560250A236BA&ipbits=0&sparams=dur,expire,id,initcwndbps,ip,ipbits,ipbypass,itag,lmt,mime,mm,mn,ms,mv,nh,pl,ratebypass,requiressl,source,upn&expire=1462191132&lmt=1462118851198277&key=cms1&title=Awesome+Gold%2FGhast+Farm+(up+to+2%2C000+tears%2F250+gold+blocks+per+hour)&redirect_counter=1&req_id=fe18baf525c5a3ee&cms_redirect=yes&ipbypass=yes&mm=31&mn=sn-vgqsen7s&ms=au&mt=1462169545&mv=m" return Content-Range='': response headers={'Content-Length': '1250', 'Content-Type': 'text/html; charset=utf-8'}, retry 1048577-2097152 May 02 17:53:30.793 - [DEBUG] ssl_closed 45.64.31.30 May 02 17:53:30.793 - [DEBUG] 45.64.31.30 worker close:no range May 02 17:53:31.217 - [DEBUG] CONNECT GET https://www.twitter.com/xxnet May 02 17:53:31.341 - [DEBUG] GET http://www.twitter.com/xxnet May 02 17:53:31.356 - [DEBUG] GET http://www.twitter.com/xxnet May 02 17:53:31.374 - [INFO] GAE CONNECT www.twitter.com:443 May 02 17:53:31.399 - [DEBUG] CONNECT GET https://www.twitter.com/xxnet May 02 17:53:31.408 - [INFO] GAE CONNECT www.twitter.com:443 May 02 17:53:31.421 - [INFO] GAE CONNECT www.twitter.com:443 May 02 17:53:31.422 - [DEBUG] GET http://www.twitter.com/xxnet May 02 17:53:31.513 - [DEBUG] GET http://www.twitter.com/xxnet May 02 17:53:31.726 - [WARNING] RangeFetch "GET https://r1---sn-vgqsen7s.googlevideo.com/videoplayback?id=o-AOiNN-GKnaxtwlYZ5rm7QTWUzaOURJkmCgpdQY-hMoDv&requiressl=yes&pl=27&ratebypass=yes&ip=107.178.194.110&source=youtube&upn=vtogaPNMlto&itag=22&mime=video/mp4&dur=1915.344&fexp=9416126,9416891,9422596,9428398,9431012,9433096,9433946,9434290&sver=3&signature=6B8FC8BE61B015724FD1D8D5A3AB5A428F459B28.2E137558F08E12A89F26B68BAB76560250A236BA&ipbits=0&sparams=dur,expire,id,initcwndbps,ip,ipbits,ipbypass,itag,lmt,mime,mm,mn,ms,mv,nh,pl,ratebypass,requiressl,source,upn&expire=1462191132&lmt=1462118851198277&key=cms1&title=Awesome+Gold%2FGhast+Farm+(up+to+2%2C000+tears%2F250+gold+blocks+per+hour)&redirect_counter=1&req_id=fe18baf525c5a3ee&cms_redirect=yes&ipbypass=yes&mm=31&mn=sn-vgqsen7s&ms=au&mt=1462169545&mv=m" return Content-Range='': response headers={'Content-Length': '1250', 'Content-Type': 'text/html; charset=utf-8'}, retry 9437185-10485760 May 02 17:53:31.731 - [DEBUG] ssl_closed 202.51.67.26 May 02 17:53:31.732 - [DEBUG] 202.51.67.26 worker close:no range May 02 17:53:31.735 - [DEBUG] 210.61.221.182 Send:HeadersFrame 15 F:END_HEADERS May 02 17:53:31.738 - [DEBUG] 210.61.221.182 Send:DataFrame 15 F:END_STREAM len:851 May 02 17:53:32.456 - [WARNING] RangeFetch "GET https://r1---sn-vgqsen7s.googlevideo.com/videoplayback?id=o-AOiNN-GKnaxtwlYZ5rm7QTWUzaOURJkmCgpdQY-hMoDv&requiressl=yes&pl=27&ratebypass=yes&ip=107.178.194.110&source=youtube&upn=vtogaPNMlto&itag=22&mime=video/mp4&dur=1915.344&fexp=9416126,9416891,9422596,9428398,9431012,9433096,9433946,9434290&sver=3&signature=6B8FC8BE61B015724FD1D8D5A3AB5A428F459B28.2E137558F08E12A89F26B68BAB76560250A236BA&ipbits=0&sparams=dur,expire,id,initcwndbps,ip,ipbits,ipbypass,itag,lmt,mime,mm,mn,ms,mv,nh,pl,ratebypass,requiressl,source,upn&expire=1462191132&lmt=1462118851198277&key=cms1&title=Awesome+Gold%2FGhast+Farm+(up+to+2%2C000+tears%2F250+gold+blocks+per+hour)&redirect_counter=1&req_id=fe18baf525c5a3ee&cms_redirect=yes&ipbypass=yes&mm=31&mn=sn-vgqsen7s&ms=au&mt=1462169545&mv=m" return Content-Range='': response headers={'Content-Length': '1250', 'Content-Type': 'text/html; charset=utf-8'}, retry 14680065-15728640 May 02 17:53:32.461 - [DEBUG] ssl_closed 66.96.226.220 May 02 17:53:32.462 - [DEBUG] 66.96.226.220 worker close:no range May 02 17:53:32.926 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:32.928 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:32.931 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:32.933 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:33.281 - [WARNING] RangeFetch "GET https://r1---sn-vgqsen7s.googlevideo.com/videoplayback?id=o-AOiNN-GKnaxtwlYZ5rm7QTWUzaOURJkmCgpdQY-hMoDv&requiressl=yes&pl=27&ratebypass=yes&ip=107.178.194.110&source=youtube&upn=vtogaPNMlto&itag=22&mime=video/mp4&dur=1915.344&fexp=9416126,9416891,9422596,9428398,9431012,9433096,9433946,9434290&sver=3&signature=6B8FC8BE61B015724FD1D8D5A3AB5A428F459B28.2E137558F08E12A89F26B68BAB76560250A236BA&ipbits=0&sparams=dur,expire,id,initcwndbps,ip,ipbits,ipbypass,itag,lmt,mime,mm,mn,ms,mv,nh,pl,ratebypass,requiressl,source,upn&expire=1462191132&lmt=1462118851198277&key=cms1&title=Awesome+Gold%2FGhast+Farm+(up+to+2%2C000+tears%2F250+gold+blocks+per+hour)&redirect_counter=1&req_id=fe18baf525c5a3ee&cms_redirect=yes&ipbypass=yes&mm=31&mn=sn-vgqsen7s&ms=au&mt=1462169545&mv=m" return Content-Range='': response headers={'Content-Length': '1250', 'Content-Type': 'text/html; charset=utf-8'}, retry 20971521-22020096 May 02 17:53:33.287 - [DEBUG] ssl_closed 202.75.141.207 May 02 17:53:33.288 - [DEBUG] 202.75.141.207 worker close:no range May 02 17:53:36.131 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:36.913 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:38.675 - [INFO] GAE CONNECT chrome-sync.sandbox.google.com:443 May 02 17:53:38.734 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:42.928 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:42.932 - [DEBUG] stream:5 frame size:16384 increase win:276380 May 02 17:53:42.933 - [DEBUG] 210.61.221.182 Send:WindowUpdateFrame 5 win_inc:276380 May 02 17:53:42.958 - [DEBUG] GAE CONNECT POST https://chrome-sync.sandbox.google.com/chrome-sync/alpha/command/?client=Google+Chrome&client_id=qI2X%2BmdCKFJSfMV2Fqirkg%3D%3D May 02 17:53:43.198 - [DEBUG] ip:118.174.27.150 real fail May 02 17:53:44.793 - [DEBUG] 210.61.221.182 Recv:DataFrame 5 len:16384 May 02 17:53:45.039 - [WARNING] RangeFetch "GET https://r1---sn-vgqsen7s.googlevideo.com/videoplayback?id=o-AOiNN-GKnaxtwlYZ5rm7QTWUzaOURJkmCgpdQY-hMoDv&requiressl=yes&pl=27&ratebypass=yes&ip=107.178.194.110&source=youtube&upn=vtogaPNMlto&itag=22&mime=video/mp4&dur=1915.344&fexp=9416126,9416891,9422596,9428398,9431012,9433096,9433946,9434290&sver=3&signature=6B8FC8BE61B015724FD1D8D5A3AB5A428F459B28.2E137558F08E12A89F26B68BAB76560250A236BA&ipbits=0&sparams=dur,expire,id,initcwndbps,ip,ipbits,ipbypass,itag,lmt,mime,mm,mn,ms,mv,nh,pl,ratebypass,requiressl,source,upn&expire=1462191132&lmt=1462118851198277&key=cms1&title=Awesome+Gold%2FGhast+Farm+(up+to+2%2C000+tears%2F250+gold+blocks+per+hour)&redirect_counter=1&req_id=fe18baf525c5a3ee&cms_redirect=yes&ipbypass=yes&mm=31&mn=sn-vgqsen7s&ms=au&mt=1462169545&mv=m" return Content-Range='': response headers={'Content-Length': '1250', 'Content-Type': 'text/html; charset=utf-8'}, retry 1048577-2097152

xxnet commented 8 years ago

​似乎是流量跑超了。

你用的是公共appid还是自己的？

821938089 commented 8 years ago

@xxnet 自己的appid

xxnet commented 8 years ago

​那你放了几个appid? 从日志看似乎是服务端返回了异常，但具体内容没有打印出来，不清楚具体原因。 我加上了，你下载master 分支 https://github.com/XX-net/XX-Net/archive/master.zip 跑下，把日志再发上来

821938089 commented 8 years ago

5个appid，现在试试下这个

821938089 commented 8 years ago

怎么存日志，刚刚跑完，结果发现web的GAEProxy日志并不能存那么长，翻上去的时候大部分日志都没了

xxnet commented 8 years ago

重新下个开发分支： ​https://github.com/XX-net/XX-Net/archive/master.zip

把http/2的debug日志关掉了。 同时可以滚屏2000行

821938089 commented 8 years ago

我大概知道是为什么了，多线程分片下载它不是按顺序的，有可能第一段下完，第二段不知道为什么没有开始下还是网络原因一直没速度下不了，然后就开始下后面的了，直到其他的下完才下这一段，导致要全部下完才能下那一段。