search

PeterDing / BaiduPCS-Py

BaiduPCS API & App 百度网盘客户端 和 API
MIT License
690 stars 113 forks source link

调用上传接口上传大于2G的文件，很大概率会13%左右出现远程主机强迫关闭了一个现有的连接。使用的最新版 #84

Closed szllzs closed 2 years ago

szllzs commented 2 years ago

WARNNING: Please to search the similar bugs, before you report a bug. Don't report a similar bug which existed. 注意：在你提交一份报告前，请先搜索是否存在类似的报告。请勿重复提交内容相同的报告。

Prerequisites / 报告前提 Before you report a bug, please let the bug to be reproduced at the latest verion of BaiduPCS-Py. 在你提交报告前，请在 BaiduPCS-Py 的最新版本上复现问题。

At sometimes, there are errors occured by an expired cookie. Before you commit the report, please to check whether cookies are expired. 有时一些错误是由于某个 cookie 超期导致的。在你提交报告前，请先检查你的 cookies 是否超期。

Describe the bug / 描述 bug A clear and concise description of what the bug is. 请清楚的描述你遇到的问题。

To Reproduce / 复现问题 Steps to reproduce the behavior:

  1. Do '...'
  2. Do '....'
  3. ...
  4. See error

按照下面的步骤可以复现问题：

  1. 做 '...'
  2. 做 '...'
  3. ...
  4. 问题出现

Screenshots / 问题截图 If applicable, add screenshots to help explain your problem.

Don't show BDUSS and cookies.

如果可能，请附加问题截图。

不要出现 BDUSS 和 cookies。

Envrionment / 运行环境

Runing log / 运行日志 Please follow steps to paste the content of file ~/.baidupcs-py/running.log.

  1. Remove the file ~/.baidupcs-py/running.log if it exists.
  2. Run the command where the bug occurs with envrionment variable LOG_LEVEL=DEBUG. e.g. LOG_LEVEL=DEBUG BaiduPCS-Py upload /abc /
  3. Paste the content of file ~/.baidupcs-py/running.log after the bug occurs.

Don't show BDUSS and cookies.

请按照下面的步骤贴出运行日志 ~/.baidupcs-py/running.log 中的内容。

  1. 删除 ~/.baidupcs-py/running.log，如果存在。
  2. 在问题发生的命令前加入环境变量 LOG_LEVEL=DEBUG。 例如：LOG_LEVEL=DEBUG BaiduPCS-Py upload /abc /
  3. 在问题出现后，贴出 ~/.baidupcs-py/running.log 中的内容。

不要出现 BDUSS 和 cookies。

Additional context / 补充内容 Add any other context about the problem here. 在这里增加补充内容。

PeterDing commented 2 years ago

你用的是哪个上传接口，上传接口有两个。同时贴一下输出的出错信息。

szllzs commented 2 years ago

接口：uploadRetPCSFile = api.upload_file(f, remotepath, callback=upload_callback)

报错堆栈： 文件上传进度：11%(279666688/2371981470) Traceback (most recent call last): File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request conn.request(method, url, httplib_request_kw) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\urllib3\connection.py", line 234, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 1257, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 1303, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 1252, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 1051, in _send_output self.send(chunk) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 973, in send self.sock.sendall(data) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\ssl.py", line 1204, in sendall v = self.send(byte_view[count:]) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\ssl.py", line 1173, in send return self._sslobj.write(data) ConnectionResetError: [WinError 10054] 远程主机强迫关闭了一个现有的连接。 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\requests\adapters.py", line 440, in send resp = conn.urlopen( File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen retries = retries.increment( File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\urllib3\util\retry.py", line 532, in increment raise six.reraise(type(error), error, _stacktrace) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\urllib3\packages\six.py", line 769, in reraise raise value.with_traceback(tb) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request conn.request(method, url, httplib_request_kw) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\urllib3\connection.py", line 234, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 1257, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 1303, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 1252, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 1051, in _send_output self.send(chunk) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 973, in send self.sock.sendall(data) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\ssl.py", line 1204, in sendall v = self.send(byte_view[count:]) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\ssl.py", line 1173, in send return self._sslobj.write(data) urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\baidupcs_py\baidupcs\pcs.py", line 197, in _request resp = self._session.request( File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\requests\sessions.py", line 529, in request resp = self.send(prep, send_kwargs) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\requests\sessions.py", line 645, in send r = adapter.send(request, kwargs) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\requests\adapters.py", line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "P:/88db8/vnr/learn/frida/project/移动懒人包/apppack/apkrepack/ps2apkrepack-noimg.py", line 401, in uploadRetPCSFile = api.upload_file(f, remotepath, callback=upload_callback) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\baidupcs_py\baidupcs\api.py", line 138, in upload_file info = self._baidupcs.upload_file( File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\baidupcs_py\baidupcs\errors.py", line 137, in check info = func(*args, **kwargs) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\baidupcs_py\baidupcs\pcs.py", line 319, in upload_file resp = self._request(Method.Post, url, params=params, data=monitor) File "C:\Users\szl\AppData\Local\Programs\Python\Python39-32\lib\site-packages\baidupcs_py\baidupcs\pcs.py", line 208, in _request raise BaiduPCSError("BaiduPCS._request", cause=err) baidupcs_py.baidupcs.errors.BaiduPCSError: BaiduPCS._request python-BaseException

PeterDing commented 2 years ago

BaiduPCSApi.upload_file 只能用于上传2G以下的文件。建议100MB以上的文件用分片上传 BaiduPCSApi.upload_slice + BaiduPCSApi.combine_slices。

大致过程如下：

from io import BytesIO
from baidupcs_py import BaiduPCSApi
from baidupcs_py.commands.upload import DEFAULT_SLICE_SIZE

api = BaiduPCSApi(...)

fd = open("some-file", "rb")

slice_md5s = []

while True:
    buf = fd.read(DEFAULT_SLICE_SIZE)
    if not buf:
        break

    md5 = api.upload_slice(BytesIO(buf))
    slice_md5s.append(md5)

api.combine_slices(slice_md5s, "/to/here")
szllzs commented 2 years ago

非常感谢