search

y-young / nazurin

🎉 Images collection done right. Telegram 图片收藏机器人 / A Telegram bot that helps you collect ACG illustrations.
http://nazurin.readthedocs.io
MIT License
264 stars 59 forks source link

Onedrive作为存储报错 Cannot create a file without content #78

Closed zsbai closed 1 year ago

zsbai commented 1 year ago

Describe the bug Onedrive作为存储，成功创建了文件夹，但是报错无法上传文件，telegram bot回复“Response Error: 400 Bad Request“

2023-06-07 10:13:21,291-0400 - INFO     - nazurin.middleware:25 - update:658614568 - Message 490: https://twitter.com/pieleaf_x2/status/1656239069461102592?s=46&t=sRguD45342wIFRmICSHhQw
2023-06-07 10:13:21,294-0400 - INFO     - nazurin.bot:123 - update:658614568 - Collection update: site=twitter, match=('1656239069461102592',)
2023-06-07 10:13:21,306-0400 - INFO     - nazurin.sites.twitter.api.web:114 - update:658614568 - Fetching tweet 1656239069461102592 from web API
2023-06-07 10:13:21,318-0400 - INFO     - aiohttp.web_log:206 -  - 127.0.0.1 "POST /923219210:AJSMNNWJDSzsdsaAOOQOPWEnkajsd HTTP/1.0" 200 152 "-" "-"
2023-06-07 10:13:23,163-0400 - INFO     - nazurin.storage.onedrive:66 - update:658614568 - Creating folders: {PosixPath('Nazurin/Twitter')}
2023-06-07 10:13:23,167-0400 - INFO     - nazurin.storage.onedrive:37 - update:658614568 - Creating drive item...
2023-06-07 10:13:24,063-0400 - ERROR    - nazurin.storage.onedrive:192 - update:658614568 - {"error":{"code":"invalidRequest","message":"Cannot create a file without content","innerError":{"date":"2023-06-07T14:11:07","request-id":"3ac0xxxxxx","client-request-id":"3ac02xxxxxx"}}}
Traceback (most recent call last):
  File "/root/nazurin/nazurin/__main__.py", line 91, in on_error
    raise exception
  File "/usr/local/lib/python3.9/dist-packages/aiogram/dispatcher/dispatcher.py", line 1394, in process_response
    response = task.result()
  File "/root/nazurin/nazurin/dispatcher.py", line 119, in update_collection
    await self.bot.update_collection(urls, message)
  File "/root/nazurin/nazurin/bot.py", line 157, in update_collection
    await self.storage.store(illust)
  File "/root/nazurin/nazurin/storage/__init__.py", line 26, in store
    await asyncio.gather(*tasks)
  File "/root/nazurin/nazurin/storage/onedrive.py", line 70, in store
    await asyncio.gather(*tasks)
  File "/root/nazurin/nazurin/storage/onedrive.py", line 49, in upload
    response = await self._request("POST", create_file_url, json=body)
  File "/usr/local/lib/python3.9/dist-packages/tenacity/_asyncio.py", line 48, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.9/dist-packages/tenacity/__init__.py", line 320, in iter
    return fut.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/dist-packages/tenacity/_asyncio.py", line 51, in call
    result = yield from fn(*args, **kwargs)
  File "/root/nazurin/nazurin/storage/onedrive.py", line 193, in _request
    response.raise_for_status()
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('https://graph.microsoft.com/v1.0/me/drive/items/root:/Nazurin/Twitter:/children')

Expected behavior A clear and concise description of what you expected to happen. 正常上传图片，然而只是创建了文件夹

Environment (please complete the following information): Linux debian 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux Python 3.9.2

Additional context 同时如果我更换了onedrive的client ID和secret，数据库并不会自己更新，一直会尝试用旧的....手动去删除了数据库中的onedrive才能使用本地.env中的新client ID和secret

y-young commented 1 year ago

有可能是因为图片没有下载成功，可以到 data/temp 里看一下，或者把 data/temp 清空再重试一次

同时如果我更换了onedrive的client ID和secret，数据库并不会自己更新，一直会尝试用旧的....手动去删除了数据库中的onedrive才能使用本地.env中的新client ID和secret

这个应该是 Access Token 没有更新，修改了 Client ID 和 Secret 之后需要手动删除

另外，你的 Bot Token 泄露了，记得重置一下

zsbai commented 1 year ago

感谢 没注意到这段日志漏token了

图片应该是下载成功了，因为我同时配置了telegram频道嘛 图片已经正常推送到频道里了

y-young commented 1 year ago

有可能是转发的消息，或者如果有用 Telegram 当存储的话看看文件有没有发送到频道里？

zsbai commented 1 year ago

有的有的 图片文件也发在频道里了 我也设置了tg存储，图片和文件都正常发送了

y-young commented 1 year ago

重试以后还是同样的报错吗？其他链接是否正常？

zsbai commented 1 year ago

重试了几次也是这样 目前测试了推特和pixiv，都出现了图片和文件正常发送到tg，但是无法上传到onedrive（相同报错）

y-young commented 1 year ago

我这边无法复现，需要更多信息

如果方便的话到这个文件里把这一行 https://github.com/y-young/nazurin/blob/16d3ca41b0c571daa1a64c0464ec65bdc55fb5c4/nazurin/storage/onedrive.py#L41

改成这样：

        size = await file.size()
        logger.info("File: {}", file)

然后重启 Bot，重试一次，把新的日志贴上来（记得把 Bot Token 923219210:AJSMNNW 删掉）

另外我看到你在 #48 也用了 OneDrive，当时是同一个网盘而且没有问题？大概是什么时候出现问题的？

zsbai commented 1 year ago 
======== Running on http://0.0.0.0:7900 ========
(Press CTRL+C to quit)
2023-06-08 00:54:01,092-0400 - INFO     - nazurin.bot:167 -  - Cleaned up temporary directory
2023-06-08 00:54:06,834-0400 - INFO     - nazurin.middleware:25 - update:658614574 - Message 504: https://twitter.com/pieleaf_x2/status/1656239069461102592
2023-06-08 00:54:06,842-0400 - INFO     - nazurin.bot:123 - update:658614574 - Collection update: site=twitter, match=('1656239069461102592',)
2023-06-08 00:54:06,855-0400 - INFO     - nazurin.sites.twitter.api.web:114 - update:658614574 - Fetching tweet 1656239069461102592 from web API
2023-06-08 00:54:06,857-0400 - INFO     - nazurin.sites.twitter.api.web:178 - update:658614574 - Fetching guest token
2023-06-08 00:54:06,865-0400 - INFO     - aiohttp.web_log:206 -  - 127.0.0.1 "POST /bot:token HTTP/1.0" 200 152 "-" "-"
2023-06-08 00:54:07,232-0400 - INFO     - nazurin.sites.twitter.api.web:188 - update:658614574 - Fetched guest token: 1666669339750301696
2023-06-08 00:54:10,584-0400 - INFO     - nazurin.storage.onedrive:132 - update:658614574 - OneDrive logged in through cached tokens
2023-06-08 00:54:10,586-0400 - INFO     - nazurin.storage.onedrive:66 - update:658614574 - Creating folders: {PosixPath('Nazurin/Twitter')}
2023-06-08 00:54:11,881-0400 - INFO     - nazurin.storage.onedrive:36 - update:658614574 - Creating drive item...
2023-06-08 00:54:11,882-0400 - INFO     - nazurin.storage.onedrive:42 - update:658614574 - File: Image(name='1656239069461102592_0 - りーふぱいザウルス(1482888205).jpg', url='https://pbs.twimg.com/media/FvwkpUZaYAAoCiw?format=jpg&name=orig', _destination='Twitter', thumbnail='https://pbs.twimg.com/media/FvwkpUZaYAAoCiw?format=jpg&name=large', _size=185140, width=938, height=1447, _chosen_url='https://pbs.twimg.com/media/FvwkpUZaYAAoCiw?format=jpg&name=orig')
2023-06-08 00:54:13,116-0400 - ERROR    - nazurin.storage.onedrive:192 - update:658614574 - {"error":{"code":"invalidRequest","message":"Cannot create a file without content","innerError":{"date":"2023-06-08T04:51:56","request-id":"c0f65677-b84d-45c6-a090-5a14f489901f","client-request-id":"c0f65677-b84d-45c6-a090-5a14f489901f"}}}
Traceback (most recent call last):
  File "/root/nazurin/nazurin/__main__.py", line 91, in on_error
    raise exception
  File "/usr/local/lib/python3.9/dist-packages/aiogram/dispatcher/dispatcher.py", line 1394, in process_response
    response = task.result()
  File "/root/nazurin/nazurin/dispatcher.py", line 119, in update_collection
    await self.bot.update_collection(urls, message)
  File "/root/nazurin/nazurin/bot.py", line 157, in update_collection
    await self.storage.store(illust)
  File "/root/nazurin/nazurin/storage/__init__.py", line 26, in store
    await asyncio.gather(*tasks)
  File "/root/nazurin/nazurin/storage/onedrive.py", line 70, in store
    await asyncio.gather(*tasks)
  File "/root/nazurin/nazurin/storage/onedrive.py", line 49, in upload
    response = await self._request("POST", create_file_url, json=body)
  File "/usr/local/lib/python3.9/dist-packages/tenacity/_asyncio.py", line 48, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.9/dist-packages/tenacity/__init__.py", line 320, in iter
    return fut.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/dist-packages/tenacity/_asyncio.py", line 51, in call
    result = yield from fn(*args, **kwargs)
  File "/root/nazurin/nazurin/storage/onedrive.py", line 193, in _request
    response.raise_for_status()
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('https://graph.microsoft.com/v1.0/me/drive/items/root:/Nazurin/Twitter:/children')

之前一直是可以正常上传的，由于之前E5大清理嘛，我的Onedrive就更换到了自己的号上，然后就出现了之前说的bot一直用旧的client ID导致启动不起来，当时在考试期所以没管，过了两个月左右到现在重新git clone了一下找不到之前的版本号了emmmm

我当时用的是很早的一个版本，当时上传图片到网盘时还没有分类（不会创建pixiv Twitter文件夹），当时刚刚修复了貌似是B站的一个bug，所以我回退到了d9a2d5c 测试，也报错了，但是报错内容貌似不一样

2023-06-08 01:06:08,469 - nazurin.bot - INFO - Collection update: site=twitter, match=('1656239069461102592',)
2023-06-08 01:06:08,485 - aiohttp.access.web_log - INFO - 127.0.0.1 [08/Jun/2023:05:06:08 +0000] "POST /bot:token HTTP/1.0" 200 152 "-" "-"
2023-06-08 01:06:11,980 - nazurin.image - INFO - Got image size: 185140
2023-06-08 01:06:12,641 - nazurin.onedrive - INFO - OneDrive logged in through cached tokens
2023-06-08 01:06:14,029 - nazurin.onedrive - INFO - OneDrive folder ID cached
2023-06-08 01:06:14,030 - nazurin.onedrive - INFO - Creating drive item...
Traceback (most recent call last):
  File "/root/nazurin-test/nazurin/__main__.py", line 56, in on_error
    raise exception
  File "/usr/local/lib/python3.9/dist-packages/aiogram/dispatcher/dispatcher.py", line 1394, in process_response
    response = task.result()
  File "/root/nazurin-test/nazurin/dispatcher.py", line 59, in update_collection
    await self.bot.update_collection(urls, message)
  File "/root/nazurin-test/nazurin/bot.py", line 128, in update_collection
    await self.storage.store(illust)
  File "/root/nazurin-test/nazurin/storage/__init__.py", line 26, in store
    await asyncio.gather(*tasks)
  File "/root/nazurin-test/nazurin/storage/onedrive.py", line 58, in store
    await asyncio.gather(*tasks)
  File "/root/nazurin-test/nazurin/storage/onedrive.py", line 43, in upload
    response = await self._request('POST', create_file_url, json=body)
  File "/usr/local/lib/python3.9/dist-packages/tenacity/_asyncio.py", line 48, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.9/dist-packages/tenacity/__init__.py", line 320, in iter
    return fut.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/dist-packages/tenacity/_asyncio.py", line 51, in call
    result = yield from fn(*args, **kwargs)
  File "/root/nazurin-test/nazurin/storage/onedrive.py", line 149, in _request
    response.raise_for_status()
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('https://graph.microsoft.com/v1.0/me/drive/items/D1E0C64C04885EBD!1970/children')
y-young commented 1 year ago

之前一直是可以正常上传的，由于之前E5大清理嘛，我的Onedrive就更换到了自己的号上，然后就出现了之前说的bot一直用旧的client ID导致启动不起来

如果更换了账号需要用个人账号在 Azure Active Directory 重新注册应用，rclone 用这套应用凭据能正常使用吗？

zsbai commented 1 year ago

可以的，我从rclone中登录，确定rclone lsd onedrive:/ 可以正常获取到onedrive内的文件夹，然后从rclone配置文件中复制相应的东西到nazurin中了。nazurin虽然没有上传图片，但是已经创建了对应site的文件夹，所以我感觉可能不是onedrive配置的问题emmm

y-young commented 1 year ago

稍微重构了一下上传的函数，可以用 refactor/onedrive-upload 分支试试

zsbai commented 1 year ago 
2023-06-08 08:28:31,554-0400 - INFO     - nazurin.models.image:68 - update:658614577 - Got image size: 185140
2023-06-08 08:28:35,362-0400 - INFO     - nazurin.storage.onedrive:169 - update:658614577 - OneDrive access token updated
2023-06-08 08:28:35,364-0400 - INFO     - nazurin.storage.onedrive:64 - update:658614577 - Creating folders: {PosixPath('Nazurin/Twitter')}
2023-06-08 08:28:36,401-0400 - INFO     - nazurin.storage.onedrive:44 - update:658614577 - Creating upload session...
2023-06-08 08:28:37,733-0400 - INFO     - nazurin.storage.onedrive:209 - update:658614577 - Uploading file, total size: 185140...
2023-06-08 08:28:38,278-0400 - INFO     - nazurin.storage.onedrive:221 - update:658614577 - Uploaded 185140
2023-06-08 08:28:38,280-0400 - INFO     - nazurin.storage.onedrive:222 - update:658614577 - Upload completed
2023-06-08 08:28:38,281-0400 - INFO     - nazurin.storage:27 - update:658614577 - Storage completed

感谢！此分支已经可以正常上传！

y-young commented 1 year ago

Fixed in https://github.com/y-young/nazurin/commit/21031894f487362219d09c7fdd4f516734d91269 .