billryan
commented
7 years ago 你可以用 -ddd 打印更详细的日志，我用 Python3 的版本也遇到类似的问题了，用 Python2 则正常。
我的可疑日志：
urllib3.exceptions.HeaderParsingError: [MissingHeaderBodySeparatorDefect()], unparsed data...Closed icehong closed 2 years ago
billryan
commented
7 years ago 你可以用 -ddd 打印更详细的日志，我用 Python3 的版本也遇到类似的问题了，用 Python2 则正常。
我的可疑日志：
urllib3.exceptions.HeaderParsingError: [MissingHeaderBodySeparatorDefect()], unparsed data...
houtianze
commented
7 years ago 感谢反馈。
这个问题好像是一直存在的。没找到解决办法。。timeout也设了，但是出现问题的时候好像没用。 用travis做build，有时候一天commit几次，前几个还能跑完测试，后面的就会出现10分钟一直没输出，build直接error的情况。所以根据这个来判断，可能是百度那边做了些限流措施之类的。以前搜过一下，据说限流的一种手段是服务器每秒只发一两个字节过来，客户端这边也不会timeout，但是很久也不会有多少进度，怀疑是这类情形。
当然，也可能是reqeusts或者http库之类的问题，不过没找到任何蛛丝马迹，我觉得是百多服务端的处理方式可能性更大。
SmartLayer
commented
7 years ago 以前搜过一下，据说限流的一种手段是服务器每秒只发一两个字节过来，客户端这边也不会timeout，但是很久也不会有多少进度，怀疑是这类情形。--> This can be verified by tcpdump, if it's really the case.
SmartLayer
commented
7 years ago I have bypyp 1.5.7 and Python 2.6. The following has been like that for 8 hours:
$ bypy -v downfile CC2017/'01 ps 2017 中英文''01 ps 2017英文版.dmg' Loading Hash Cache File '/home/gulliver/.bypy/bypy.hashcache.json'... Hash Cache File loaded. [==__] 10% (160.0MB/1.5GB) ETA: 5h35m (67kB/s, 40m33s gone)
The progress bar only moved in 1 hour, then in the next 8 hours, the ETA remains (should change to 'forever').
houtianze
commented
7 years ago --> This can be verified by tcpdump, if it's really the case.
Thanks for the information. Could you show me how to do that? Or can you verify at your end since you are having this problem?
houtianze
commented
7 years ago The progress bar only moved in 1 hour, then in the next 8 hours, the ETA remains (should change to 'forever').
The progress bar only updates once a block (default is 20MB if i'm not wrong) has been transferred, so if the network is throttled, it will take a prohibitive long period to update.
satouriko
commented
6 years ago 可否在客户端这边设置超时多长时间自动重传
houtianze
commented
6 years ago 好像有个参数—timeout还是什么的可以控制超时，但是。。这个卡住的好像没用，很可能是百度服务端的限速。
On Mon, 19 Mar 2018 at 2:02 PM, LJH notifications@github.com wrote:
可否在客户端这边设置超时多长时间自动重传
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/houtianze/bypy/issues/373#issuecomment-374109695, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGC5schGL4VygRavVdvEtG1RXSyfFdIks5tf0oCgaJpZM4N2-GR .
satouriko
commented
6 years ago 我的意思是说 设置一个时间 如果超过这个时间一个 block 还没有传完，就重新开始传（效果上和 kill 掉重开一样）
另外我这边往往是第一次卡住会超时 然后重试的时候就永久卡死了。今天更新版本以后才这样，以前是重试虽然一直不好使但是 n*5 次以后就又开始传了
Keywords [ラブライブ] hit
任务日志
<D> [13:41:06] Using requests.packages.urllib3 to disable warnings
<D> [13:41:06] Using urllib3 to disable warnings
Loading Hash Cache File '/root/.bypy/bypy.hashcache.json'...
Hash Cache File loaded.
----
Verbose level = 1
Debug level = True
Config directory: '/root/.bypy'
Token file: '/root/.bypy/bypy.json'
Hash Cache file: '/root/.bypy/bypy.hashcache.json'
App root path at Baidu Yun '/apps/bypy'
sys.stdin.encoding = UTF-8
sys.stdout.encoding = UTF-8
sys.stderr.encoding = UTF-8
----
<D> Token loaded:
<D> {'refresh_token': '22.a5675e6a8f8fd204e6784e7ca48f6d74.315360000.1835612658.30021648-1572671', 'expires_in': 2592000, 'session_key': '9mnRIE9SIKBcUE8Dd5nAijgOFc3P47jgY+MeEwHl2hJqm0KGlKLmjwu1YjRG++/TIldtw3oZAX5ayobuVyx4RG5oIP928SY=', 'scope': 'basic netdisk', 'access_token': '21.dd4a7a9c3f8a547cf2fbf9ef2e2a13d9.2592000.1522844658.30021648-1572671', 'session_secret': 'd471543cb369d2c849c183ed439defa4'}
<D> Uploading file '【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv'
<D> __get_file_info(): rdir : /apps/bypy/nijigasaki | rfile: 【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> GET https://pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Params: {'method': 'list', 'order': 'asc', 'by': 'name', 'path': '/apps/bypy/nijigasaki'}
<D> Full URL: https://pcs.baidu.com/rest/2.0/pcs/file?method=list&path=%2Fapps%2Fbypy%2Fnijigasaki&by=name&order=asc&access_token=21.dd4a7a9c3f8a547cf2fbf9ef2e2a13d9.2592000.1522844658.30021648-1572671
<D> HTTP Status Code: 200
<D> 200 OK, processing action
<D> List json: {'list': [{'md5': '7c9e3e5b5084c0c350c365d6835f914a', 'mtime': 1520348333, 'size': 670469807, 'isdir': 0, 'fs_id': 624580931781532, 'ctime': 1520348333, 'path': '/apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第5回.mkv'}, {'md5': '635084cdc6f997d1b54b64c32c1ed4ac', 'mtime': 1520593204, 'size': 400912134, 'isdir': 0, 'fs_id': 676235449855768, 'ctime': 1520593204, 'path': '/apps/bypy/nijigasaki/【虹ヶ咲学園スクールアイドル同好会】スクフェス分室動画企画「メンバー紹介！エマ編」.webm'}], 'request_id': 1799072318476555547}
<D> remote path is '/apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv'
<D> '【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' is being RapidUploaded.
<D> [13:41:15] Cache miss for file '【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv',
md5: 8dd9b1a971ce59b828812b2e495c7ee2
size: 867683488
mtime: 1521093454
<D> [13:41:15] Periodically saving Hash Cash
<D> [13:41:15] Cache miss for file '【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv',
slice_md5: d66cf6ab0e74570860c5dee5855967f4
size: 867683488
mtime: 1521093454
<D> [13:41:15] Periodically saving Hash Cash
<D> [13:41:23] Cache miss for file '【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv',
crc32: 772561114
size: 867683488
mtime: 1521093454
<D> [13:41:23] Periodically saving Hash Cash
<D> RapidUploading Length: 867683488 MD5: 8dd9b1a971ce59b828812b2e495c7ee2, Slice-MD5: d66cf6ab0e74570860c5dee5855967f4, CRC: 0x2e0c58da
<D> POST https://pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: None
<D> Params: {'content-length': 867683488, 'ondup': 'overwrite', 'slice-md5': 'd66cf6ab0e74570860c5dee5855967f4', 'method': 'rapidupload', 'content-crc32': '0x2e0c58da', 'path': '/apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv', 'content-md5': '8dd9b1a971ce59b828812b2e495c7ee2'}
<D> Full URL: https://pcs.baidu.com/rest/2.0/pcs/file?content-length=867683488&ondup=overwrite&access_token=21.dd4a7a9c3f8a547cf2fbf9ef2e2a13d9.2592000.1522844658.30021648-1572671&slice-md5=d66cf6ab0e74570860c5dee5855967f4&method=rapidupload&content-crc32=0x2e0c58da&path=%2Fapps%2Fbypy%2Fnijigasaki%2F%E3%80%90%E3%83%A9%E3%83%96%E3%83%A9%E3%82%A4%E3%83%96%EF%BC%81%E3%82%B9%E3%82%AF%E3%82%B9%E3%82%BF%E3%80%91%E7%9B%B8%E8%89%AF%E8%8C%89%E5%84%AA%E3%80%81%E4%B9%85%E4%BF%9D%E7%94%B0%E6%9C%AA%E5%A4%A2%E3%80%81%E6%A5%A0%E6%9C%A8%E3%81%A8%E3%82%82%E3%82%8A%E3%81%AE%E6%94%BE%E8%AA%B2%E5%BE%8C%E3%83%81%E3%83%A3%E3%83%AC%E3%83%B3%E3%82%B8%E7%AC%AC6%E5%9B%9E.mkv&content-md5=8dd9b1a971ce59b828812b2e495c7ee2
<D> HTTP Status Code: 404
<D> Website returned JSON: {'error_code': 31079, 'request_id': 1799076931242354780, 'error_msg': 'file md5 not found, you should use upload api to upload the whole file.'}
<D> MD5 not found, rapidupload failed
<D> '【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' can't be RapidUploaded, now trying normal uploading.
<D> '【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' is being slicing uploaded.
<D> Slice size: 20971520, Pieces: 42
<D> Find the progress entry resume uploading
<D> Verifying the md5s. Total count = 2
<D> b5f60c3923384df167d5fe074f97050f verified
<D> 0f11c135af1aee1fbad1b8eb4ab93db2 verified
<D> verified md5 count = 2
<D> Start from offset 41943040
<D> Uploading MD5 slice: f7d005e2126b09e6b9e95e36d850ba1e, #3 / 42
<D> POST https://c.pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Params: {'type': 'tmpfile', 'method': 'upload'}
<D> Full URL: https://c.pcs.baidu.com/rest/2.0/pcs/file?type=tmpfile&method=upload&access_token=21.dd4a7a9c3f8a547cf2fbf9ef2e2a13d9.2592000.1522844658.30021648-1572671
<D> HTTP Status Code: 200
<D> 200 OK, processing action
<D> Uploaded MD5 slice: f7d005e2126b09e6b9e95e36d850ba1e
'【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' >>==> '/apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' OK.
<D> Request OK
<D> Slice MD5 match, continuing next slice
<D> Updating slice upload progress for /root/go/src/github.com/cool2645/youtube-to-netdisk/static/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Uploading MD5 slice: a84a9e72308778605b071060f14e0376, #4 / 42
<D> POST https://c.pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Params: {'type': 'tmpfile', 'method': 'upload'}
<D> Full URL: https://c.pcs.baidu.com/rest/2.0/pcs/file?type=tmpfile&method=upload&access_token=21.dd4a7a9c3f8a547cf2fbf9ef2e2a13d9.2592000.1522844658.30021648-1572671
<D> HTTP Status Code: 200
<D> 200 OK, processing action
<D> Uploaded MD5 slice: a84a9e72308778605b071060f14e0376
'【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' >>==> '/apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' OK.
<D> Request OK
<D> Slice MD5 match, continuing next slice
<D> Updating slice upload progress for /root/go/src/github.com/cool2645/youtube-to-netdisk/static/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Uploading MD5 slice: f18f217f08799be4df784eefb46492fd, #5 / 42
<D> POST https://c.pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Params: {'type': 'tmpfile', 'method': 'upload'}
<D> Full URL: https://c.pcs.baidu.com/rest/2.0/pcs/file?type=tmpfile&method=upload&access_token=21.dd4a7a9c3f8a547cf2fbf9ef2e2a13d9.2592000.1522844658.30021648-1572671
<D> HTTP Status Code: 200
<D> 200 OK, processing action
<D> Uploaded MD5 slice: f18f217f08799be4df784eefb46492fd
'【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' >>==> '/apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' OK.
<D> Request OK
<D> Slice MD5 match, continuing next slice
<D> Updating slice upload progress for /root/go/src/github.com/cool2645/youtube-to-netdisk/static/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Uploading MD5 slice: 0bf9c3d51dd715a3fc24c9c9ea88e5ed, #6 / 42
<D> POST https://c.pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Params: {'type': 'tmpfile', 'method': 'upload'}
<D> Full URL: https://c.pcs.baidu.com/rest/2.0/pcs/file?type=tmpfile&method=upload&access_token=21.dd4a7a9c3f8a547cf2fbf9ef2e2a13d9.2592000.1522844658.30021648-1572671
<D> HTTP Status Code: 200
<D> 200 OK, processing action
<D> Uploaded MD5 slice: 0bf9c3d51dd715a3fc24c9c9ea88e5ed
'【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' >>==> '/apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' OK.
<D> Request OK
<D> Slice MD5 match, continuing next slice
<D> Updating slice upload progress for /root/go/src/github.com/cool2645/youtube-to-netdisk/static/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Uploading MD5 slice: 55987d3e2ce324789e830b6fba8deb8c, #7 / 42
<D> POST https://c.pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Params: {'type': 'tmpfile', 'method': 'upload'}
<D> Full URL: https://c.pcs.baidu.com/rest/2.0/pcs/file?type=tmpfile&method=upload&access_token=21.dd4a7a9c3f8a547cf2fbf9ef2e2a13d9.2592000.1522844658.30021648-1572671
<D> HTTP Status Code: 200
<D> 200 OK, processing action
<D> Uploaded MD5 slice: 55987d3e2ce324789e830b6fba8deb8c
'【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' >>==> '/apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv' OK.
<D> Request OK
<D> Slice MD5 match, continuing next slice
<D> Updating slice upload progress for /root/go/src/github.com/cool2645/youtube-to-netdisk/static/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Uploading MD5 slice: 64254c94d6a20cfafa9ec865c7a706e8, #8 / 42
<D> POST https://c.pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Params: {'type': 'tmpfile', 'method': 'upload'}
<E> [14:14:22] Error accessing 'https://c.pcs.baidu.com/rest/2.0/pcs/file'
<E> [14:14:22] Exception:
<class 'requests.exceptions.ConnectionError'> - ('Connection aborted.', TimeoutError(110, 'Connection timed out'))
Stack:
File "../syncBaidu.py", line 20, in <module>
bp.upload(fn, folder + fn)
File "/usr/local/lib/python3.4/dist-packages/bypy/bypy.py", line 1759, in upload
return self.__upload_file(lpath, rpath, ondup)
File "/usr/local/lib/python3.4/dist-packages/bypy/bypy.py", line 1706, in __upload_file
result = self.__upload_file_slices(localpath, remotepath, ondup)
File "/usr/local/lib/python3.4/dist-packages/bypy/bypy.py", line 1506, in __upload_file_slices
ec = self.__upload_slice(remotepath)
File "/usr/local/lib/python3.4/dist-packages/bypy/bypy.py", line 1429, in __upload_slice
pars, self.__upload_slice_act, remotepath)
File "/usr/local/lib/python3.4/dist-packages/bypy/bypy.py", line 1421, in __stream_upload
headers=headers, data=form)
File "/usr/local/lib/python3.4/dist-packages/bypy/bypy.py", line 858, in __post
return self.__request(url, pars, act, 'POST', actargs, retry, addtoken, dumpex, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/bypy/bypy.py", line 833, in __request
result = self.__request_work(url, pars, act, method, actargs, addtoken, dumpex, **kwnew)
File "/usr/local/lib/python3.4/dist-packages/bypy/bypy.py", line 787, in __request_work
self.__dump_exception(ex, url, pars, r, act)
File "/usr/local/lib/python3.4/dist-packages/bypy/bypy.py", line 602, in __dump_exception
perr(formatex(ex))
File "/usr/local/lib/python3.4/dist-packages/bypy/util.py", line 73, in formatex
type(ex), ex, ''.join(traceback.format_stack()))
<E> [14:14:22] Function: __upload_slice_act
<E> [14:14:22] Website parameters: {'type': 'tmpfile', 'method': 'upload'}
<E> [14:14:22] Waiting 10 seconds before retrying...
<E> [14:14:32] Request Try #2 / 5
<D> POST https://c.pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /apps/bypy/nijigasaki/【ラブライブ！スクスタ】相良茉優、久保田未夢、楠木ともりの放課後チャレンジ第6回.mkv
<D> Params: {'type': 'tmpfile', 'method': 'upload'}
[=___________________] 7% (60.0MB/827.5MB)
[=___________________] 9% (80.0MB/827.5MB)
[==__________________] 12% (100.0MB/827.5MB) ETA: 1h18s (206kB/s, 4m58s gone)
[==__________________] 14% (120.0MB/827.5MB) ETA: 56m16s (215kB/s, 6m21s gone)
[===_________________] 16% (140.0MB/827.5MB) ETA: 50m52s (231kB/s, 7m23s gone) 看你贴出的log，并没有卡死，应该是重试一次后就继续下载了。 重试就是应对timeout这个问题的。
satouriko
commented
6 years ago 但是它现在、比如现在我这个还在运行着，卡住19%，已经7个小时没有输出了（趴）
Hou Tianze notifications@github.com 于 2018年3月19日周一 22:51写道：
看你贴出的log，并没有卡死，应该是重试一次后就继续下载了。 重试就是应对timeout这个问题的。
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/houtianze/bypy/issues/373#issuecomment-374239518, or mute the thread https://github.com/notifications/unsubscribe-auth/ARha8QcX1BDCEez3FcWi8qkudQrop5qFks5tf8XzgaJpZM4N2-GR .
houtianze
commented
6 years ago 这个好像是百度限流，想不到什么好办法。下载的话你可以试试--downloader aria2调用aria2来下载看是否有改进。
以下是带的日志， 看看还需要什么其它的日志信息
需要说明的是， 我是在 screen 命令中执行的bypy命令（防止终端关闭退出）， 同时，把日志 用 tee 命令写了文件 ， 不知道和这两个命令有没有关系
日志信息就停在这个地方， 也看不出来什么异常。