【漏洞】VideoUploader CHUNK_FAILED

liaozd commented 1 year ago

Python 版本： 3.10

模块版本： 15.1.0

运行环境： Ubuntu

模块路径： bilibili_api.video_uploader.VideoUploader

解释器： cpython / pypi / jython

报错信息： {'name': 'PREUPLOAD', 'data': {<bilibili_api.video_uploader.VideoUploaderPage object at 0x7f32eb52b670>: <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f32eb52b670>}} {'name': 'PRE_PAGE', 'data': {'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f32eb52b670>}} {'name': 'PRE_CHUNK', 'data': {'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f32eb52b670>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 3}} {'name': 'PRE_CHUNK', 'data': {'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f32eb52b670>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 3}} {'name': 'PRE_CHUNK', 'data': {'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f32eb52b670>, 'offset': 20971520, 'chunk_number': 2, 'total_chunk_count': 3}} {'name': 'CHUNK_FAILED', 'data': {'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f32eb52b670>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 3, 'info': ''}} {'name': 'CHUNK_FAILED', 'data': {'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f32eb52b670>, 'offset': 20971520, 'chunk_number': 2, 'total_chunk_count': 3, 'info': ''}} {'name': 'CHUNK_FAILED', 'data': {'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f32eb52b670>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 3, 'info': ''}}

这里填写报错信息

报错代码： try exception 出现WriteTimout异常

{'timeout': {'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}}

这里填写报错代码

在此处写正文

Nemo2011 commented 1 year ago

settings.timeout 试一下，不行的话直接设置 httpx 默认 timeout

liaozd commented 1 year ago

settings.timeout 试一下，不行的话直接设置 httpx 默认 timeout

settings.timeout 试过了，应该是没有效果

Nemo2011 commented 1 year ago

可以了，原来是 bug，你可以用 dev 分支代码试一下

hazelement commented 1 year ago

我这也遇到同样的问题,我用dev试了也不行.credential没有问题,可以like video

{'name': 'PRE_PAGE', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 20971520, 'chunk_number': 2, 'total_chunk_count': 4},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 20971520, 'chunk_number': 2, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 31457280, 'chunk_number': 3, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 4},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 31457280, 'chunk_number': 3, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 20971520, 'chunk_number': 2, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7fa576421840>, 'offset': 31457280, 'chunk_number': 3, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data':

git log

commit 183a01eac92ef6a225823364018f8711cf611011 (HEAD -> dev, origin/main, origin/dev, origin/HEAD, main)
Author: Nemo2011 <yimoxia@outlook.com>
Date:   Wed Mar 8 20:24:37 2023 +0800

    fix!: Game.get_comments 失效

my code

    metadata = {
        "act_reserve_create": 0,
        "copyright": 1,
        "source": "",
        "desc": "",
        "desc_format_id": 0,
        "dynamic": "",
        "interactive": 0,
        "no_reprint": 1,
        "open_elec": 0,
        "origin_state": 0,
        "subtitles": {
            "lan": "",
            "open": 0
        },
        "tag": "周杰伦",
        "tid": 137,
        "title": video_title,
        "up_close_danmaku": False,
        "up_close_reply": False,
        "up_selection_reply": False,
    }
    uploader = VideoUploader([p], metadata,
                             credential,
                             cover=video_file_path.replace('.mp4', '.png')
                             )
    print(f"Uploading [{video_file_path}] with title [{video_title}]")
    print(f"metadata: [{metadata}]")

    @uploader.on("__ALL__")
    async def ev(data):
        print(data)
    await uploader.start()

Nemo2011 commented 1 year ago

能展示一下全部源代码吗

hazelement commented 1 year ago

from unicodedata import name
from bilibili_api import sync, video
from bilibili_api.video_uploader import VideoUploaderPage, VideoUploader
from bilibili_api.video_zone import get_zone_videos_count_today
from bilibili_api.utils.Credential import Credential
from bilibili_api import settings
import bilibili_api
settings.timeout = 10.0

async def upload_video(video_file_path, video_title):

    p = VideoUploaderPage(video_file_path, video_title, video_title)
    credential = Credential(...)

    print(bilibili_api.BILIBILI_API_VERSION)

    metadata = {
        "act_reserve_create": 0,
        "copyright": 1,
        "source": "",
        "desc": "",
        "desc_format_id": 0,
        "dynamic": "",
        "interactive": 0,
        "no_reprint": 1,
        "open_elec": 0,
        "origin_state": 0,
        "subtitles": {
            "lan": "",
            "open": 0
        },
        "tag": "周杰伦",
        "tid": 137,
        "title": video_title,
        "up_close_danmaku": False,
        "up_close_reply": False,
        "up_selection_reply": False,
    }
    uploader = VideoUploader([p], metadata,
                             credential,
                             cover=video_file_path.replace('.mp4', '.png')
                             )
    print(f"Uploading [{video_file_path}] with title [{video_title}]")
    print(f"metadata: [{metadata}]")

    @uploader.on("__ALL__")
    async def ev(data):
        print(data)
    await uploader.start()

if __name__ == '__main__':
    sync(upload_video('2023-03-05.mp4', 'test title'))

output

-> % python upload.py                 
15.3.1
Uploading [2023-03-05.mp4] with title [test title]
metadata: [{'act_reserve_create': 0, 'copyright': 1, 'source': '', 'desc': '', 'desc_format_id': 0, 'dynamic': '', 'interactive': 0, 'no_reprint': 1, 'open_elec': 0, 'origin_state': 0, 'subtitles': {'lan': '', 'open': 0}, 'tag': '周杰伦', 'tid': 137, 'title': 'test title', 'up_close_danmaku': False, 'up_close_reply': False, 'up_selection_reply': False}]
{'name': 'PREUPLOAD', 'data': ({<bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>: <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>},)}
{'name': 'PRE_PAGE', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 20971520, 'chunk_number': 2, 'total_chunk_count': 4},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 20971520, 'chunk_number': 2, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 31457280, 'chunk_number': 3, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 4},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 31457280, 'chunk_number': 3, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 4, 'info': ''},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 20971520, 'chunk_number': 2, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 31457280, 'chunk_number': 3, 'total_chunk_count': 4},)}
{'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f0be6ad02e0>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 4},)}

Nemo2011 commented 1 year ago

你要不先看一下具体报错原因，参考这里的做法，增加报错代码

https://github.com/Nemo2011/bilibili-api/issues/210

liaozd commented 1 year ago

bilibili-api-python==15.3.1，升级了最新版本，不过好像不大行

`Connected to pydev debugger (build 223.8836.34) {'name': 'PREUPLOAD', 'data': ({<bilibili_api.video_uploader.VideoUploaderPage object at 0x7f2c4ae82f20>: <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f2c4ae82f20>},)} {'name': 'PRE_PAGE', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f2c4ae82f20>},)} {'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f2c4ae82f20>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 3},)} {'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f2c4ae82f20>, 'offset': 10485760, 'chunk_number': 1, 'total_chunk_count': 3},)} {'name': 'PRE_CHUNK', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f2c4ae82f20>, 'offset': 20971520, 'chunk_number': 2, 'total_chunk_count': 3},)} {'name': 'CHUNK_FAILED', 'data': ({'page': <bilibili_api.video_uploader.VideoUploaderPage object at 0x7f2c4ae82f20>, 'offset': 0, 'chunk_number': 0, 'total_chunk_count': 3, 'info': ''},)} {'name': 'FAILED', 'data': ({'err': WriteTimeout('')},)} python-BaseException Traceback (most recent call last): File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 51, in write await self._stream.send(item=buffer) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/anyio/streams/tls.py", line 202, in send await self._call_sslobject_method(self._ssl_object.write, item) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/anyio/streams/tls.py", line 168, in _call_sslobject_method await self.transport_stream.send(self._write_bio.read()) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1307, in send await self._protocol.write_event.wait() File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait await fut asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions yield File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 50, in write with anyio.fail_after(timeout): File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/anyio/_core/_tasks.py", line 118, in exit raise TimeoutError TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions yield File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request raise exc File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request response = await connection.handle_async_request(request) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request return await self._connection.handle_async_request(request) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/_async/http11.py", line 112, in handle_async_request raise exc File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/_async/http11.py", line 82, in handle_async_request await self._send_request_body(**kwargs) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/_async/http11.py", line 135, in _send_request_body await self._send_event(event, timeout=timeout) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/_async/http11.py", line 144, in _send_event await self._network_stream.write(bytes_to_send, timeout=timeout) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 49, in write with map_exceptions(exc_map): File "/usr/lib/python3.10/contextlib.py", line 153, in exit self.gen.throw(typ, value, traceback) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) httpcore.WriteTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/liaozhuodi/git-repos/tunnel/scripts/debug/debug_downloader.py", line 56, in main await uploader.start() File "/home/liaozhuodi/git-repos/others/_python_projects/bilibili-api/bilibili_api/video_uploader.py", line 452, in start raise e File "/home/liaozhuodi/git-repos/others/_python_projects/bilibili-api/bilibili_api/video_uploader.py", line 444, in start result = await task File "/home/liaozhuodi/git-repos/others/_python_projects/bilibili-api/bilibili_api/video_uploader.py", line 412, in _main data = await self._upload_page(page) File "/home/liaozhuodi/git-repos/others/_python_projects/bilibili-api/bilibili_api/video_uploader.py", line 514, in _upload_page result = await asyncio.gather(*tasks) File "/home/liaozhuodi/git-repos/others/_python_projects/bilibili-api/bilibili_api/video_uploader.py", line 640, in _upload_chunk raise e File "/home/liaozhuodi/git-repos/others/_python_projects/bilibili-api/bilibili_api/video_uploader.py", line 611, in _upload_chunk resp = await session.put( File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_client.py", line 1885, in put return await self.request( File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request return await self.send(request, auth=auth, follow_redirects=follow_redirects) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_client.py", line 1620, in send response = await self._send_handling_auth( File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_client.py", line 1648, in _send_handling_auth response = await self._send_handling_redirects( File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects response = await self._send_single_request(request) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_client.py", line 1722, in _send_single_request response = await transport.handle_async_request(request) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request with map_httpcore_exceptions(): File "/usr/lib/python3.10/contextlib.py", line 153, in exit self.gen.throw(typ, value, traceback) File "/home/liaozhuodi/.cache/pypoetry/virtualenvs/tunnel-p6VXeO5R-py3.10/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.WriteTimeout

Process finished with exit code 1`

Nemo2011 commented 1 year ago

干脆粗暴一点，上传时设置timeout久一点，如果还是还是还是不行的话直接去设置httpx的默认timeout吧，看看是不是模块的问题。

hazelement commented 1 year ago

check out my pull request,

https://github.com/Nemo2011/bilibili-api/pull/234

z0z0r4 commented 1 year ago

可以 close 了

Nemo2011 / bilibili-api

【漏洞】VideoUploader CHUNK_FAILED #226