search

yzfedora / baidudl

This is a multi-thread download tool for pan.baidu.com
GNU General Public License v3.0
198 stars 35 forks source link

The baidudl doesn't work anymore. #12

Open soyoeml opened 6 years ago

soyoeml commented 6 years ago

Please take a look my debug information:

bdpandl -d 5 -n 5 "https://d11.baidupcs.com/file/ce3c40d6636a176351c15fb3b99dd88a?bkt=p3-0000e991d91cc454a73dcc7117ef07edcd24&xcode=f19376c6ef1b92cc9ef1bae760d2a1c2c64cc6201111a512230e8240066c6ae634fd712cebb40050&fid=3532746553-250528-679694564704771&time=1522172136&sign=FDTAXGERLQBHSKa-DCb740ccc5511e5e8fedcff06b081203-m%2Fv%2Bd3Hsy3RvEBXfoJhK28p79Uo%3D&to=d11&size=3768805274&sta_dx=3768805274&sta_cs=63202&sta_ft=mp4&sta_ct=5&sta_mt=1&fm2=MH%2CYangquan%2CAnywhere%2C%2Cnew_york%2Cany&vuk=1496566989&iv=0&newver=1&newfm=1&secfm=1&flow_ver=3&pkey=0000e991d91cc454a73dcc7117ef07edcd24&sl=76480590&expires=8h&rt=sh&r=248414905&mlogid=1996122626872874054&vbdid=3723150705&fin=%E7%8B%97%E5%8D%81%E4%B8%89.mp4&fn=%E7%8B%97%E5%8D%81%E4%B8%89.mp4&rtype=1&dp-logid=1996122626872874054&dp-callid=0.1.1&hps=1&tsl=80&csl=80&csign=LvjJ57Fc%2BFmI9uEePVIqRlqnxQA%3D&so=0&ut=6&uter=4&serv=1&uc=2643518180&ic=3351836320&ti=bfb42dc18c68bf069c6fbe6bd649483d63fc02c6ec817aaf&by=themis" --------------Received Meta info--------------- HTTP/1.1 302 Moved Temporarily Server: bfe/1.0.8.13-sslpool-patch Date: Tue, 27 Mar 2018 18:03:31 GMT Content-Type: text/html Content-Length: 160 Connection: keep-alive Location: http://www.baidupcs.com/403.html Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS, HEAD

--------------Received Meta info--------------- HTTP/1.1 403 Forbidden Date: Tue, 27 Mar 2018 18:03:31 GMT Content-Type: text/html Content-Length: 62 Connection: close ETag: "5566be1c-3e" Server: POMS/CloudUI 1.0

--------------Received Meta info--------------- HTTP/1.1 302 Moved Temporarily Server: bfe/1.0.8.13-sslpool-patch Date: Tue, 27 Mar 2018 18:03:32 GMT Content-Type: text/html Content-Length: 160 Connection: keep-alive Location: http://www.baidupcs.com/403.html Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS, HEAD

--------------Received Meta info--------------- HTTP/1.1 403 Forbidden Date: Tue, 27 Mar 2018 18:03:33 GMT Content-Type: text/html Content-Length: 62 Connection: keep-alive ETag: "5566be1c-3e" Server: POMS/CloudUI 1.0

failed to download from url: https://d11.baidupcs.com/file/ce3c40d6636a176351c15fb3b99dd88a?bkt=p3-0000e991d91cc454a73dcc7117ef07edcd24&xcode=f19376c6ef1b92cc9ef1bae760d2a1c2c64cc6201111a512230e8240066c6ae634fd712cebb40050&fid=3532746553-250528-679694564704771&time=1522172136&sign=FDTAXGERLQBHSKa-DCb740ccc5511e5e8fedcff06b081203-m%2Fv%2Bd3Hsy3RvEBXfoJhK28p79Uo%3D&to=d11&size=3768805274&sta_dx=3768805274&sta_cs=63202&sta_ft=mp4&sta_ct=5&sta_mt=1&fm2=MH%2CYangquan%2CAnywhere%2C%2Cnew_york%2Cany&vuk=1496566989&iv=0&newver=1&newfm=1&secfm=1&flow_ver=3&pkey=0000e991d91cc454a73dcc7117ef07edcd24&sl=76480590&expires=8h&rt=sh&r=248414905&mlogid=1996122626872874054&vbdid=3723150705&fin=%E7%8B%97%E5%8D%81%E4%B8%89.mp4&fn=%E7%8B%97%E5%8D%81%E4%B8%89.mp4&rtype=1&dp-logid=1996122626872874054&dp-callid=0.1.1&hps=1&tsl=80&csl=80&csign=LvjJ57Fc%2BFmI9uEePVIqRlqnxQA%3D&so=0&ut=6&uter=4&serv=1&uc=2643518180&ic=3351836320&ti=bfb42dc18c68bf069c6fbe6bd649483d63fc02c6ec817aaf&by=themis

yzfedora commented 6 years ago

@soyoeml Yes, I noticed this problem in one month ago, This problem is caused by baidu.com, they added the cookies authentication to the download server I think. if you just copy the download link, it will no longer works. I have no plan to add cookies or login supports to this tool so far. this tool should works like "axel". but supports extra https protocol.

You can have a look at this tool: https://github.com/GangZhuo/BaiduPCS, It's still working.

soyoeml commented 6 years ago

But I still like yours. Please don't waste previous work and keep updating. Thanks

yzfedora commented 6 years ago

@soyoeml I added a few commits today for fix the pan.baidu.com download URL not longer working bug, can you try latest git version and confirm it does fixed the problem? thanks

soyoeml commented 6 years ago

I tested and it has different errors. Please see the logs:

bdpandl -d 5 -n 0 "https://d.pcs.baidu.com/file/436898de4954433ad950c0d1d396e601?fid=1656127312-250528-683387019251627&dstime=1524513920&rt=sh&sign=FDtAERVY-DCb740ccc5511e5e8fedcff06b081203-pqnnWIP2AEt6lfOvPB300KuMI%2BE%3D&expires=8h&chkv=1&chkbd=0&chkpc=et&dp-logid=2624740452660832607&dp-callid=0&r=561662504#0" --------------Received Meta info--------------- HTTP/1.1 302 Moved Temporarily Date: Mon, 23 Apr 2018 20:11:19 GMT Content-Type: text/plain; charset=utf-8 Connection: keep-alive x-bs-client-ip: MjQuNC43LjI= x-bs-request-id: Ymp5ei1jbG91ZC1zdG9yYWdlLWYxMm5ldzA4NS5ianl6LmJhaWR1LmNvbToxMC4xNTEuMzguMTk6ODA5ODoyNjI0NzQwNDUyNjYwODMyNjA3OjIwMTgtMDQtMjQgMDQ6MTE6MTk= Content-Length: 51 x-pcs-member: 0 Remote-Ip: pcs.bae.baidu.com Location: https://d11.baidupcs.com/file/436898de4954433ad950c0d1d396e601?bkt=p3-0000e16db40685ed32672f2284264334fad8&xcode=191e2e998b2122c0caf46d8089c08cb8d3e7a5125933195540e47162580dafa874e07f4a95a3cd95&fid=1656127312-250528-683387019251627&time=1524514279&sign=FDTAXGERQBHSKa-DCb740ccc5511e5e8fedcff06b081203-oCky%2F6UJssdgiu4WzsyTDe%2F16gU%3D&to=d11&size=1112646592&sta_dx=1112646592&sta_cs=9352&sta_ft=mp41&sta_ct=0&sta_mt=0&fm2=MH%2CYangquan%2CAnywhere%2C%2Cnew_jersey%2Cany&vuk=282335&iv=0&newver=1&newfm=1&secfm=1&flow_ver=3&pkey=0000e16db40685ed32672f2284264334fad8&expires=8h&rt=sh&r=561662504&mlogid=2624740452660832607&vbdid=-&fin=XBSJ.S02E01.720p-%E5%A4%A9%E5%A4%A9%E7%BE%8E%E5%89%A7%E5%AD%97%E5%B9%95%E7%BB%84.mp41&fn=XBSJ.S02E01.720p-%E5%A4%A9%E5%A4%A9%E7%BE%8E%E5%89%A7%E5%AD%97%E5%B9%95%E7%BB%84.mp41&rtype=1&dp-logid=2624740452660832607&dp-callid=0.1.1&hps=1&tsl=0&csl=0&csign=I4eYot93%2F9FxiT9ssVUgylihUK0%3D&so=0&ut=6&uter=4&serv=0&uc=1171098929&ic=905446300&ti=f8fdaa4589ff2f18c4a23adfdd7e61a6b24b9b4b011c7f03&by=themis x-poms-key: 0000e16db40685ed32672f2284264334fad8 Server: POMS/CloudUI 1.0

--------------Received Meta info--------------- HTTP/1.1 200 OK Server: bfe/1.0.8.13-sslpool-patch Date: Mon, 23 Apr 2018 20:11:20 GMT Content-Type: application/octet-stream Content-Length: 1112646592 Connection: keep-alive Last-Modified: Mon, 23 Apr 2018 07:49:15 GMT Expires: Thu, 26 Apr 2018 08:18:59 GMT Age: 42741 Cache-Control: max-age=259200 Accept-Ranges: bytes x-bs-file-size: 1112646592 x-bs-meta-crc32: 0 x-bs-client-ip: MTExLjE3Ny4zLjEzNw== x-bs-request-id: MTAuMjE0LjIyMy4zMzo4NjQzOjI2MTI5NjAwMTE3MTc3NjU4OTk6MjAxOC0wNC0yMyAxNTo1Mzo1OQ== Content-Disposition: attachment;filename=XBSJ.S02E01.720p-%E5%A4%A9%E5%A4%A9%E7%BE%8E%E5%89%A7%E5%AD%97%E5%B9%95%E7%BB%84.mp41 superfile: 2 Ohc-Response-Time: 1 0 0 0 0 0 Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS, HEAD

filename=XXXX.S02E01.720p.mp41, length=1112646592

thread 0 starting to download range: 0-1112646591 XXXX.S02E01.720p.mp41 1.0GiB 0.00% 0.00KiB/s 12 days [1]remote server may not supports multithreading download XXXX.S02E01.720p.mp41 1.0GiB 0.00% 0.00KiB/s 12 days [1]Segmentation fault

yzfedora commented 6 years ago

@soyoeml I noticed you are using option "-n 0", the default value is "-n 100", means 100 threads for download. if you set it to 0 I am not sure what will happens. can you just try "bdpandl -d 1 -n 100 ..."

soyoeml commented 6 years ago

@yzfedora Even I set -n 5. It has same error.

yzfedora commented 6 years ago

@soyoeml I confirmed if you set "-n 0", and it will segment fault. but I can run with "-n 5" option, and without any problem. only download speed slow. If you still get segment fault. you can use gdb to find the segment fault place in 1 minute.

$ gdb /usr/bin/bdpandl
and in the gdb, type:
$ run -d 5 -n 5 "...."
after segment fault happens, type "bt" command. and check the stack information.
soyoeml commented 6 years ago 
[New Thread 0x4887f480 (LWP 5774)]
thread 0 starting to download range: 0-195725009
[New Thread 0x4907f480 (LWP 5775)]
thread 1 starting to download range: 195725010-391450019
[New Thread 0x4987f480 (LWP 5776)]
thread 2 starting to download range: 391450020-587175029
[New Thread 0x4a07f480 (LWP 5777)]
thread 3 starting to download range: 587175030-782900039
[New Thread 0x4a87f480 (LWP 5778)]
thread 4 starting to download range: 782900040-978625052
xxxxxxx.mkv                          933.3MiB     0.00%  0.00KiB/s   11 days  [5]remote server may not supports multithreading download
remote server may not supports multithreading download
remote server may not supports multithreading download
remote server may not supports multithreading download
remote server may not supports multithreading download
XXXXXX.mkv                          933.3MiB     0.00%  0.00KiB/s   11 days  [5]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x4887f480 (LWP 5774)]
0x100066ec in dlbuffer_write ()
(gdb) bt
#0  0x100066ec in dlbuffer_write ()
#1  0x10005378 in dlpart_write_callback ()
#2  0x0ff57dfc in ?? () from /usr/lib/libcurl.so.4
#3  0x0ff71d84 in ?? () from /usr/lib/libcurl.so.4
#4  0x0ff720bc in ?? () from /usr/lib/libcurl.so.4
#5  0x0ff72eec in curl_easy_perform () from /usr/lib/libcurl.so.4
#6  0x10005624 in dlpart_launch ()
#7  0x100044b4 in dlinfo_download ()
#8  0x0ffc6528 in start_thread () from /lib/libpthread.so.0
#9  0x0fc6e6f0 in clone () from /lib/libc.so.6
yzfedora commented 6 years ago

@soyoeml which Linux distribution are you using? it looks you are using a 32-bit system, right?

soyoeml commented 6 years ago 
MyBookLive:~# uname -a
Linux MyBookLive 2.6.32.11-svn70860 #1 Thu May 17 13:32:51 PDT 2012 ppc GNU/Linux

Yes, it's a 32bit system.

I recompiled it. And it doesn't download one byte. But no segmentation fault any more.

bdpandl -n 1 -d 5 "https://d.pcs.baidu.com/file/5a6bc358737791a724d7c3fb5b84276c?fid=4221785630-250528-706510566481316&dstime=1524861284&rt=sh&sign=FDtAERVY-DCb740ccc5511e5e8fedcff06b081203-EuUSgUN0eX1i2SEJjklc2hlASTs%3D&expires=8h&chkv=1&chkbd=0&chkpc=et&dp-logid=2717985396274277352&dp-callid=0&r=203845362#0"
--------------Received Meta info---------------
HTTP/1.1 302 Moved Temporarily
Date: Fri, 27 Apr 2018 20:47:37 GMT
Content-Type: text/plain; charset=utf-8
Connection: keep-alive
x-bs-client-ip: MjQuNC43LjI=
x-bs-request-id: eXEwMS1vYmplY3QwMi1yMzAtMDMtMDA1LnlxMDEuYmFpZHUuY29tOjEwLjYxLjE0MC4zNDoyMDEwOjI3MTc5ODUzOTYyNzQyNzczNTI6MjAxOC0wNC0yOCAwNDo0NzozNw==
Content-Length: 51
x-pcs-member: 0
Remote-Ip: pcs.bae.baidu.com
Location: https://d11.baidupcs.com/file/5a6bc358737791a724d7c3fb5b84276c?bkt=p3-00002b54dfea5c0a3805e1a93fc3515fd69f&xcode=eaf11a9ee05c2e23fe7598da6f21729f6ae0ac783f58cc493ec56efab2821fc9ec836700211e485c&fid=4221785630-250528-706510566481316&time=1524862057&sign=FDTAXGERQBHSKa-DCb740ccc5511e5e8fedcff06b081203-577kLCxUQNGe2cRtP6oR7iuz6ek%3D&to=d11&size=978625053&sta_dx=978625053&sta_cs=347&sta_ft=mkv&sta_ct=1&sta_mt=1&fm2=MH%2CYangquan%2CAnywhere%2C%2CNone%2Cany&vuk=282335&iv=0&newver=1&newfm=1&secfm=1&flow_ver=3&pkey=00002b54dfea5c0a3805e1a93fc3515fd69f&expires=8h&rt=sh&r=203845362&mlogid=2717985396274277352&vbdid=-&fin=%5Bttmeiju.vip%5DThe.Expanse.S03E03.720p.HDTV.x264-SVA.mkv&fn=%5Bttmeiju.vip%5DThe.Expanse.S03E03.720p.HDTV.x264-SVA.mkv&rtype=1&dp-logid=2717985396274277352&dp-callid=0.1.1&hps=1&tsl=0&csl=0&csign=I4eYot93%2F9FxiT9ssVUgylihUK0%3D&so=0&ut=6&uter=4&serv=1&uc=1171098929&ic=905446300&ti=9cf67ec2c4eb6cc0f07ea5334d0e44faf72dacd9ce9e4087&by=themis
x-poms-key: 00002b54dfea5c0a3805e1a93fc3515fd69f
Server: POMS/CloudUI 1.0

--------------Received Meta info---------------
HTTP/1.1 200 OK
Server: bfe/1.0.8.13-sslpool-patch
Date: Fri, 27 Apr 2018 20:47:38 GMT
Content-Type: application/octet-stream
Content-Length: 978625053
Connection: keep-alive
ETag: 5a6bc358737791a724d7c3fb5b84276c
Last-Modified: Thu, 26 Apr 2018 02:52:07 GMT
Expires: Sun, 29 Apr 2018 04:07:14 GMT
Age: 146424
Cache-Control: max-age=259200
Accept-Ranges: bytes
x-bs-file-size: 978625053
x-bs-meta-crc32: 4217737859
x-bs-client-ip: MTExLjE3Ny4zLjEyOQ==
x-bs-request-id: MTAuMjE1LjEzNy42NTo4NjQzOjI2Nzg4NzQ5NDA4Njg5ODE3MDA6MjAxOC0wNC0yNiAxMjowNzoxMg==
Content-Disposition: attachment;filename=%5Bttmeiju.vip%5DThe.Expanse.S03E03.720p.HDTV.x264-SVA.mkv
Content-MD5: 5a6bc358737791a724d7c3fb5b84276c
superfile: 2
Ohc-Response-Time: 1 0 0 0 0 0
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS, HEAD

filename=xxx.mkv, length=978625053

thread 0 starting to download range: 0-978625052
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]remote server may not supports multithreading download
xxx.mkv                            933.3MiB     0.00%  0.00KiB/s   11 days  [1]
yzfedora commented 6 years ago

@soyoeml "anti hotlinking"... basically, you triggered baidupcs's protect mechanism, so it refused you to download I think, I get this error sometimes too, the only solution is specify 100 or more threads to download, and let the program to continue retry.

soyoeml commented 6 years ago

@yzfedora Still has segmentation fault error. It cannot download even one byte.

Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
xxxx.mp41                                        1.2GiB     0.00%  0.00KiB/s   14 days  [100]Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102
Received json error: error_code=31326, error_msg=anti hotlinking , hitcode:102

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x4a07f480 (LWP 5623)]
0x0fc081c0 in free () from /lib/libc.so.6
(gdb) bt
#0  0x0fc081c0 in free () from /lib/libc.so.6
#1  0x0fdcc844 in CRYPTO_free () from /lib/libcrypto.so.0.9.7
Backtrace stopped: frame did not save the PC