syncdown/downdir HTTP 403 : file is not authorized

N46AN commented 9 years ago

syncdown 和 downdir 出现如下问题： (如果是用户的低级错误请原谅) syncup 没有问题。

$ ./bypy.py -dddv downdir /San\ Andreas\ \(2015\)\ [1080p] /home/user/bypy
Token file: '/home/user/.bypy/bypy.json'
Hash Cache file: '/home/user/.bypy/bypy.pickle'
App root path at Baidu Yun '/apps/bypy'
sys.stdin.encoding = UTF-8
sys.stdout.encoding = None
sys.stderr.encoding = UTF-8
Verbose level = 1
Debug = 3

Loading Hash Cache File '/home/user/.bypy/bypy.pickle'...
Hash Cache File loaded.
<D> Token loaded:
<D> {u'access_token': u'21.415410603367155a35dff5e41fbd69f7.2592000.1447129276.692384115-1572671', u'expires_in': 2592000, u'session_secret': u'8119b202bfe915fb4d8781020982007c', u'scope': u'basic netdisk', u'session_key': u'9mnRcAejF2eVZTt31cT7qYq5a02R3/06ZxGgmjlXUGmX4cT9VlPDHpufrdMClUoIr9S2RosfWjFAFQwAye8ZHPgvRCrV5kde', u'refresh_token': u'22.6f89ab888b6014e5b3d396afcc933ee8.315360000.1759897276.692384115-1572671'}
<D> Error loading BDUSS:
<D> Traceback (most recent call last):
  File "./bypy.py", line 1537, in __load_local_bduss
    with open(BDUSSPath, 'rb') as infile:
IOError: [Errno 2] No such file or directory: u'/home/user/.bypy/bypy.bduss'

BDUSS not found at '/home/user/.bypy/bypy.bduss'.
<D> GET https://pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: ([], [])
<D> Params: {u'path': u'/apps/bypy/San Andreas (2015) [1080p]', u'order': u'asc', u'method': u'list', u'by': u'name'}
send: 'GET /rest/2.0/pcs/file?access_token=21.415410603367155a35dff5e41fbd69f7.2592000.1447129276.692384115-1572671&path=%2Fapps%2Fbypy%2FSan+Andreas+%282015%29+%5B1080p%5D&method=list&by=name&order=asc HTTP/1.1\r\nHost: pcs.baidu.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Access-Control-Allow-Origin: *
header: Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS, HEAD
header: x-pcs-request-id: MTAuNTcuMTI0Ljc1OjgwODA6NzQxNDk2OTIzOjExL09jdC8yMDE1IDEyOjM3OjI0IA==
header: Server: nginx/1.4.2
header: Date: Sun, 11 Oct 2015 04:37:24 GMT
header: Content-Type: text/html;charset=utf8
header: PCS-CACHE-FLAG: 1
header: x-bs-client-ip: MjIzLjIwLjE0MC4xMjk=
header: Content-Length: 258
<D> HTTP Status Code: 200
<D> Request OK, processing action
<D> Request all goes fine
<D> Remote dirs: []
<D> Remote files: [{u'isdir': 0, u'ctime': 1444538043, u'fs_id': 1065662160327483L, u'mtime': 1444538064, u'path': u'/Apps/bypy/San Andreas (2015) [1080p]/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4', u'md5': u'66b69a3721b03338afd980a07e4f82a5', u'size': 1979242212}]
<D> Downloading '/Apps/bypy/San Andreas (2015) [1080p]/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4' as '/home/user/bypy/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4'
<D> Getting info of remote file '/Apps/bypy/San Andreas (2015) [1080p]/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4' for later verification
<D> __get_file_info(): rdir : /Apps/bypy/San Andreas (2015) [1080p] | rfile: San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4
<D> GET https://pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /Apps/bypy/San Andreas (2015) [1080p]/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4
<D> Params: {u'path': u'/Apps/bypy/San Andreas (2015) [1080p]', u'order': u'asc', u'method': u'list', u'by': u'name'}
send: 'GET /rest/2.0/pcs/file?access_token=21.415410603367155a35dff5e41fbd69f7.2592000.1447129276.692384115-1572671&path=%2FApps%2Fbypy%2FSan+Andreas+%282015%29+%5B1080p%5D&method=list&by=name&order=asc HTTP/1.1\r\nHost: pcs.baidu.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Access-Control-Allow-Origin: *
header: Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS, HEAD
header: x-pcs-request-id: MTAuNTcuMTIzLjM4OjgwODA6MjYwMzgxNTU5NToxMS9PY3QvMjAxNSAxMjozNzoyNCA=
header: Server: nginx/1.4.2
header: Date: Sun, 11 Oct 2015 04:37:24 GMT
header: Content-Type: text/html;charset=utf8
header: x-bs-client-ip: MjIzLjIwLjE0MC4xMjk=
header: Content-Length: 81
<D> HTTP Status Code: 403
<E> [12:37:24] Error accessing 'https://pcs.baidu.com/rest/2.0/pcs/file'
None

<E> [12:37:24] Function: __get_file_info_act
<E> [12:37:24] Website parameters: {u'path': u'/Apps/bypy/San Andreas (2015) [1080p]', u'order': u'asc', u'method': u'list', u'by': u'name'}
<E> [12:37:24] HTTP Response Status Code: 403
<E> [12:37:24] Error code: 31064
Error Description: file is not authorized
<E> [12:37:24] Website returned: {"error_code":31064,"error_msg":"file is not authorized","request_id":2603815595}
Not saving Hash Cache since 'dirty' is 'False' and 'force_saving' is 'False'

houtianze commented 9 years ago

谢谢反馈。

貌似是百度PCS远程文件列表API有点问题，返回的文件是大写的/Apps/...开头，而不是/apps/...，这样再用这个路径去请求就会出错了。

你可以帮我试一下：把这个函数def __get_file_info(self, remotefile, **kwargs):第一行： rdir, rfile = posixpath.split(remotefile) 改成

rdir, rfile = posixpath.split(remotefile)
rfile = '/a' + rfile[2:]

看一看出的错误是不是不同了？

N46AN commented 9 years ago

是否需要改的是 rdir？改成这样：

    def __get_file_info(self, remotefile, **kwargs):
        rdir, rfile = posixpath.split(remotefile)
        print rdir
        rdir = '/a' + rdir[2:]
        print rdir

的确返回修改成功：

/Apps/bypy/San Andreas (2015) [1080p]
/apps/bypy/San Andreas (2015) [1080p]

但是错误依旧： http://paste.ubuntu.com/12748340/

N46AN commented 9 years ago

我看第一次出现错误是这里：

INFO:urllib3.connectionpool:Starting new HTTPS connection (1): d.pcs.baidu.com
send: 'GET /rest/2.0/pcs/file?ru=MC0yMDk3MTUxOQ%3D%3D&access_token=21.415410603367155a35dff5e41fbd69f7.2592000.1447129276.692384115-1572671&app_id=250528&ec=1&check_blue=1&path=%2FApps%2Fbypy%2FSan+Andreas+%282015%29+%5B1080p%5D%2FSan.Andreas.2015.1080p.BluRay.x264.YIFY.mp4&method=download HTTP/1.1\r\nHost: d.pcs.baidu.com\r\nRange: bytes=0-20971519\r\nAccept-Encoding: gzip, deflate\r\nConnection: keep-alive\r\nAccept: */*\r\nUser-Agent: netdisk;5.2.7.2;PC;PC-Windows;6.2.9200;WindowsBaiduYunGuanJia\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'

如果要怀疑路径，此处 /Apps 仍为大写不懂编程，勿怪

N46AN commented 9 years ago

ok. 按照路径大小写思路，尝试修改2660行 __proceed_downdir(self, remotepath, dirjs, filejs, args)

for filej in filejs:
            rfile = filej['path']
            relfile = rfile[rootlen:]
            #lfile = os.path.join(localpath, relfile)
            lfile = joinpath(localpath, relfile)
            print 'AAAAAAAAAAAAAAAAAAAA', rfile
            rfile = '/a' + rfile[2:]
            print 'BBBBBBBBBBBBBBBBBBBB', rfile
            self.__downfile(rfile, lfile)

这回一个大写 /Apps 都没有了，也不报 403 了。但不下载？像dry-run一样。搞不懂了...

AAAAAAAAAAAAAAAAAAAA /Apps/bypy/San Andreas (2015) [1080p]/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4
BBBBBBBBBBBBBBBBBBBB /apps/bypy/San Andreas (2015) [1080p]/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4
<D> Downloading '/apps/bypy/San Andreas (2015) [1080p]/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4' as '/home/simon/bypy/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4'
<D> Getting info of remote file '/apps/bypy/San Andreas (2015) [1080p]/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4' for later verification
/apps/bypy/San Andreas (2015) [1080p]
/apps/bypy/San Andreas (2015) [1080p]
<D> __get_file_info(): rdir : /apps/bypy/San Andreas (2015) [1080p] | rfile: San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4
<D> GET https://pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: /apps/bypy/San Andreas (2015) [1080p]/San.Andreas.2015.1080p.BluRay.x264.YIFY.mp4
<D> Params: {u'path': u'/apps/bypy/San Andreas (2015) [1080p]', u'order': u'asc', u'method': u'list', u'by': u'name'}
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): pcs.baidu.com
send: 'GET /rest/2.0/pcs/file?access_token=21.415410603367155a35dff5e41fbd69f7.2592000.1447129276.692384115-1572671&path=%2Fapps%2Fbypy%2FSan+Andreas+%282015%29+%5B1080p%5D&method=list&by=name&order=asc HTTP/1.1\r\nHost: pcs.baidu.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'

完整输出 http://paste.ubuntu.com/12748881/

N46AN commented 9 years ago

Loading Hash Cache File '/home/user/.bypy/bypy.pickle'...
Hash Cache File loaded.
<D> Token loaded:
<D> {u'access_token': u'21.415410603367155a35dff5e41fbd69f7.2592000.1447129276.692384115-1572671', u'expires_in': 2592000, u'session_secret': u'8119b202bfe915fb4d8781020982007c', u'scope': u'basic netdisk', u'session_key': u'9mnRcAejF2eVZTt31cT7qYq5a02R3/06ZxGgmjlXUGmX4cT9VlPDHpufrdMClUoIr9S2RosfWjFAFQwAye8ZHPgvRCrV5kde', u'refresh_token': u'22.6f89ab888b6014e5b3d396afcc933ee8.315360000.1759897276.692384115-1572671'}
<D> Error loading BDUSS:
<D> Traceback (most recent call last):
  File "./bypy.py", line 1537, in __load_local_bduss
    with open(BDUSSPath, 'rb') as infile:
IOError: [Errno 2] No such file or directory: u'/home/user/.bypy/bypy.bduss'

...难道一直是验证有问题么？... 重新验证了一遍，有同样问题。

houtianze commented 9 years ago

对，我弄错了，第一个地方确实应该改rdir，不是rfile。那个bduss的错误没有关系的，不会影响。你这几个改动加上之后还是下载不了？其他的list和upload命令能用吗？

houtianze commented 9 years ago

哦，还有个地方要改，第1755行： if f['path'] == remotefile: 改成： if f['path'].lower() == remotefile.lower():

总体感觉是暂时问题，百度那边估计部署了有点问题的代码，按理过几天应该会改正。

N46AN commented 9 years ago

是的，list / upload 都正常。 $ ./bypy.py -dddv downdir /San\ Andreas\ $2015$\ [1080p] /home/user/bypy 输出 http://paste.ubuntu.com/12748881/ $./bypy.py -dddv downfile /minecraft.7z /home/user/bypy 输出 http://paste.ubuntu.com/12748951/

从现在开始可能是我使用的低级错误了，但是一脑子还想不出来... :-(

N46AN commented 9 years ago

if f['path'].lower() == remotefile.lower(): 改过这个后正常了！开始以250 KiB/s 的龟速下载。（我的运营商问题）

非常感谢你的工作！问题解决了。 http://img.vim-cn.com/b8/5b5751f3cdd84ec76c98b8936b477ca2e06c88.png

houtianze commented 9 years ago

不客气，解决就好。希望百度赶快改了这个bug，要不然估计其他人也会受影响。

N46AN commented 9 years ago

LastEdit 2015-10-12 18:58 CST

今天又发现两层目录时会出问题，貌似 __get_file_info() 没有运行，先前对大小写的改动只有部分生效。第二次发送 GET 时又变成大写 /Apps 了。我先研究一下。是否这只是我个人的问题？不会所有用户都有此问题吧。

目录结构：/apps/bypy/test1/test2/test.mkv

怎么感觉 __proceed_downdir() 中 filejs 是空的？ __proceed_downdir() 改成

    def __proceed_downdir(self, remotepath, dirjs, filejs, args):
        print "__proceed_downdir is called."
        result = ENoError
        rootrpath, localpath = args
        rlen = len(remotepath) + 1 # '+ 1' for the trailing '/', it bites.
        rootlen = len(rootrpath) + 1 # ditto

        result = self.__prepare_local_dir(localpath)
        if result != ENoError:
            perr("Fail to create prepare local directory '{}' for downloading, ABORT".format(localpath))
            return result

        print "__proceed_downdir is called. for dirj in dirjs, dirjs is", dirjs
        for dirj in dirjs:
            reldir = dirj['path'][rlen:]
            #ldir = os.path.join(localpath, reldir)
            ldir = joinpath(localpath, reldir)
            result = self.__prepare_local_dir(ldir)
            if result != ENoError:
                perr("Fail to create prepare local directory '{}' for downloading, ABORT".format(ldir))
                return result

        print "__proceed_downdir is called. for filej in filejs, filejs is", filejs
        for filej in filejs:
            print "for filej in filejs, iteration", filej['path']
            rfile = filej['path']
            relfile = rfile[rootlen:]
            #lfile = os.path.join(localpath, relfile)
            lfile = joinpath(localpath, relfile)
            print 'AAAAAAAAAAAAAAAAAAAA__proceed_downdir_UPRCASE', rfile
            rfile = '/a' + rfile[2:]
            print 'BBBBBBBBBBBBBBBBBBBB__proceed_downdir_LWRCASE', rfile
            self.__downfile(rfile, lfile)

        return result

输出

__proceed_downdir is called.
__proceed_downdir is called. for dirj in dirjs, dirjs is [{u'isdir': 1, u'ctime': 1444646540, u'fs_id': 175317641805962L, u'mtime': 1444646540, u'path': u'/Apps/bypy/test1/test2', u'md5': u'', u'size': 0}]
__proceed_downdir is called. for filej in filejs, filejs is []

完整输出

$ ./bypy.py -dddv downdir /test1 /home/aaaaaa/bypy
Token file: '/home/aaaaaa/.bypy/bypy.json'
Hash Cache file: '/home/aaaaaa/.bypy/bypy.pickle'
App root path at Baidu Yun '/apps/bypy'
sys.stdin.encoding = UTF-8
sys.stdout.encoding = None
sys.stderr.encoding = UTF-8
Verbose level = 1
Debug = 3
----

Loading Hash Cache File '/home/aaaaaa/.bypy/bypy.pickle'...
Hash Cache File not found, no caching
<D> Token loaded:
<D> {u'access_token': u'21.37a9af8c8e10369cf5880c06610dcdba.2592000.1447141187.692384115-1572671', u'expires_in': 2592000, u'session_secret': u'cd8139ffac47d38dda40fbcb6869d7f8', u'scope': u'basic netdisk', u'session_key': u'9mnRdydUehRdaHDDcQJGHZLmqP1FKvEf0clGJPl89SEF0dkbYL9ED2MfHH9KzvQ1rcdVErH99MIUvTOxmd9jOlzQvGlzzEKa', u'refresh_token': u'22.86c3aebbc03a4fccb49f16c8ff2f96e0.315360000.1759909187.692384115-1572671'}
<D> Error loading BDUSS:
<D> Traceback (most recent call last):
  File "./bypy.py", line 1537, in __load_local_bduss
    with open(BDUSSPath, 'rb') as infile:
IOError: [Errno 2] No such file or directory: u'/home/aaaaaa/.bypy/bypy.bduss'

BDUSS not found at '/home/aaaaaa/.bypy/bypy.bduss'.
<D> GET https://pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: ([], [])
<D> Params: {u'path': u'/apps/bypy/test1', u'order': u'asc', u'method': u'list', u'by': u'name'}
send: 'GET /rest/2.0/pcs/file?access_token=21.37a9af8c8e10369cf5880c06610dcdba.2592000.1447141187.692384115-1572671&path=%2Fapps%2Fbypy%2Ftest1&method=list&by=name&order=asc HTTP/1.1\r\nHost: pcs.baidu.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Access-Control-Allow-Origin: *
header: Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS, HEAD
header: x-pcs-request-id: MTAuNTcuMjA3LjYwOjgwODA6NDAzMTgxNjY3NToxMi9PY3QvMjAxNSAxODo1MzowMiA=
header: Server: nginx/1.4.2
header: Date: Mon, 12 Oct 2015 10:53:02 GMT
header: Content-Type: text/html;charset=utf8
header: PCS-CACHE-FLAG: 1
header: x-bs-client-ip: MjIzLjIwLjE0MC4xMjk=
header: Content-Length: 158
<D> HTTP Status Code: 200
<D> Request OK, processing action
<D> Request all goes fine
<D> Remote dirs: [{u'isdir': 1, u'ctime': 1444646540, u'fs_id': 175317641805962L, u'mtime': 1444646540, u'path': u'/Apps/bypy/test1/test2', u'md5': u'', u'size': 0}]
<D> Remote files: []
__proceed_downdir is called.
__proceed_downdir is called. for dirj in dirjs, dirjs is [{u'isdir': 1, u'ctime': 1444646540, u'fs_id': 175317641805962L, u'mtime': 1444646540, u'path': u'/Apps/bypy/test1/test2', u'md5': u'', u'size': 0}]
__proceed_downdir is called. for filej in filejs, filejs is []
<D> GET https://pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: ([], [])
<D> Params: {u'path': u'/Apps/bypy/test1/test2', u'order': u'asc', u'method': u'list', u'by': u'name'}
send: 'GET /rest/2.0/pcs/file?access_token=21.37a9af8c8e10369cf5880c06610dcdba.2592000.1447141187.692384115-1572671&path=%2FApps%2Fbypy%2Ftest1%2Ftest2&method=list&by=name&order=asc HTTP/1.1\r\nHost: pcs.baidu.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Access-Control-Allow-Origin: *
header: Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS, HEAD
header: x-pcs-request-id: MTAuNTcuMTI1LjE4OjgwODA6MTIzNjAxMzg4NjoxMi9PY3QvMjAxNSAxODo1MzowMiA=
header: Server: nginx/1.4.2
header: Date: Mon, 12 Oct 2015 10:53:02 GMT
header: Content-Type: text/html;charset=utf8
header: x-bs-client-ip: MjIzLjIwLjE0MC4xMjk=
header: Content-Length: 81
<D> HTTP Status Code: 403
<E> [18:53:02] Error accessing 'https://pcs.baidu.com/rest/2.0/pcs/file'
None

<E> [18:53:02] Function: __walk_remote_dir_act
<E> [18:53:02] Website parameters: {u'path': u'/Apps/bypy/test1/test2', u'order': u'asc', u'method': u'list', u'by': u'name'}
<E> [18:53:02] HTTP Response Status Code: 403
<E> [18:53:02] Error code: 31064
Error Description: file is not authorized
<E> [18:53:02] Website returned: {"error_code":31064,"error_msg":"file is not authorized","request_id":1236013886}
<D> Remote dirs: []
<D> Remote files: []
<D> Error: 31064 while sub-walking remote dirs'[{u'isdir': 1, u'ctime': 1444646540, u'fs_id': 175317641805962L, u'mtime': 1444646540, u'path': u'/Apps/bypy/test1/test2', u'md5': u'', u'size': 0}]'
Not saving Hash Cache since 'dirty' is 'False' and 'force_saving' is 'False'

houtianze commented 9 years ago

估计还是哪个地方（可能是__walk_remote_dir()）没处理，我改天再看看。我还没试过，不知道是不是现在百度返回都变成这样了，如果这样就得改改代码了。

N46AN commented 9 years ago

@houtianze 谢谢，我也研究一下不懂编程，要看懂这些代码不太容易 -_-||

感觉百度的产品质量不行啊...

N46AN commented 9 years ago

@houtianze 临时修改了__walk_remote_dir_recur()，两层目录也正常了

非常感谢帮助！

修改为

def __walk_remote_dir_recur(self, remotepath, proceed, remoterootpath, args = None, skip_remote_only_dirs = False):
        print "__walk_remote_dir_recur, UPRCASE remotepath:",remotepath
        remotepath = '/a' + remotepath[2:]
        print "__walk_remote_dir_recur, LWRCASE remotepath:",remotepath
...

输出

__proceed_downdir is called.
...
__walk_remote_dir_recur, UPRCASE remotepath: /Apps/bypy/test1/test2
__walk_remote_dir_recur, LWRCASE remotepath: /apps/bypy/test1/test2
<D> GET https://pcs.baidu.com/rest/2.0/pcs/file
<D> actargs: ([], [])
<D> Params: {u'path': u'/apps/bypy/test1/test2', u'order': u'asc', u'method': u'list', u'by': u'name'}
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): pcs.baidu.com
send: 'GET /rest/2.0/pcs/file?access_token=21.37a9af8c8e10369cf5880c06610dcdba.2592000.1447141187.692384115-1572671&path=%2Fapps%2Fbypy%2Ftest1%2Ftest2&method=list&by=name&order=asc HTTP/1.1\r\n
Host: pcs.baidu.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'

houtianze commented 9 years ago

搞定啦？不错 :+1:

考虑把这个相关改动做一个临时分支，如果百度那边一直这样的话。

houtianze commented 8 years ago

做了一个临时分支（后来看改动应该只需要一行，在__walk_remote_dir_act），如果这个大写的/Apps问题还存在，应该可以解决： 1d804e2edddc43ac5c1fe515b7ee6376b5ab72c2 https://github.com/houtianze/bypy/tree/temp

houtianze / bypy

syncdown/downdir HTTP 403 : file is not authorized #185