dogbutcat / gclone

fork from https://github.com/donwa/gclone
166 stars 28 forks source link

Why All Server account had reach 403: User rate limit exceeded? #25

Closed tyhunter closed 1 year ago

tyhunter commented 1 year ago

Hi 作者,我遇到了一个问题,希望你帮忙看下。 我是使用qbittorrent搭配gclone上传到Google Drive作为离线下载使用,准备了100个SA账户,理论上每天可以上传75TB的数据,这种模式运行了快两年,一直保持较为稳定的状态,但从上周五开始,我发现大量的SA账户出现了403: User rate limit exceeded报错提示,通过debug查看到程序一直在切换SA账户,但似乎每个账户都被403限制了,所以想问下这个是不是Google修改了SA的规则限制导致的? Hi author, I have encountered a problem and I hope you can help me. I use qbittorrent with gclone to upload to Google Drive for offline download. I have prepared 100 SA accounts. Theoretically, I can upload 75TB of data every day. This mode has been running for almost two years and has remained relatively stable. However, since the Starting from Friday, I found that a large number of SA accounts had 403: User rate limit exceeded error messages. Through debugging, I found that the program has been switching SA accounts, but it seems that each account is restricted by 403, so I want to ask if this is the case. Is it caused by Google revising SA’s rules and restrictions?

2023-09-17 10:10:49 DEBUG : pacer: Rate limited, increasing sleep to 2.289498139s 2023-09-17 10:10:49 DEBUG : pacer: Reducing sleep to 0s 2023-09-17 10:10:51 DEBUG : Changing Service Account File from /root/accounts/f3a08d7f80ba7da5c612e27b4d21b285acb67ff3.json to /root/accounts/2cd2466b973ee66b999e33c6600ccfa7f65b0289.json 2023-09-17 10:10:51 NOTICE: loading gclone sa file: /root/accounts/2cd2466b973ee66b999e33c6600ccfa7f65b0289.json 2023-09-17 10:10:51 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded) 2023-09-17 10:10:51 DEBUG : pacer: Rate limited, increasing sleep to 1.840841498s 2023-09-17 10:10:51 DEBUG : Changing Service Account File from /root/accounts/2cd2466b973ee66b999e33c6600ccfa7f65b0289.json to /root/accounts/beb9581e8401519f028dcfe1794bf4bef3978ee0.json 2023-09-17 10:10:51 DEBUG : Torrent : Received error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded - low level retry 4/10 2023-09-17 10:10:51 NOTICE: loading gclone sa file: /root/accounts/beb9581e8401519f028dcfe1794bf4bef3978ee0.json 2023-09-17 10:10:51 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded) 2023-09-17 10:10:51 DEBUG : pacer: Rate limited, increasing sleep to 2.915264993s 2023-09-17 10:10:52 DEBUG : pacer: Reducing sleep to 0s 2023-09-17 10:10:53 DEBUG : Changing Service Account File from /root/accounts/beb9581e8401519f028dcfe1794bf4bef3978ee0.json to /root/accounts/3d4e2d242e75236950bf4c5e4c0a922a20ffeaf3.json 2023-09-17 10:10:53 NOTICE: loading gclone sa file: /root/accounts/3d4e2d242e75236950bf4c5e4c0a922a20ffeaf3.json 2023-09-17 10:10:53 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded) 2023-09-17 10:10:53 DEBUG : pacer: Rate limited, increasing sleep to 1.488134634s 2023-09-17 10:10:53 DEBUG : Torrent : Received error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded - low level retry 5/10 2023-09-17 10:10:53 DEBUG : Changing Service Account File from /root/accounts/3d4e2d242e75236950bf4c5e4c0a922a20ffeaf3.json to /root/accounts/896c80c1ed8f6f79bc1f289a67ae21162eeab333.json 2023-09-17 10:10:53 NOTICE: loading gclone sa file: /root/accounts/896c80c1ed8f6f79bc1f289a67ae21162eeab333.json 2023-09-17 10:10:53 DEBUG : pacer: low level retry 5/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded) 2023-09-17 10:10:53 DEBUG : pacer: Rate limited, increasing sleep to 2.325750342s

dogbutcat commented 1 year ago

您好 感谢提问 请问这100个SA账户是否都属于一个project下? 检查Google Cloud Console中该项目是否出现警告? Google Cloud Console中的API请求数量是否过大?

您描述的场景我也一样在用 目前运行没有问题 但我在去年5月出过一个事:由于qb的请求切换频率过高 似乎触发了Google的预警 直接导致project被封 官方给出的理由是一个api只能对应一个SA账号 超出请删除多余账号后再重新申请 但似乎直到我删到只剩一个再申请恢复项目一样没有通过 而且我司的某项目使用多SA方案也被干掉过 所以不清楚您的场景是否触发了官方的隐藏预警?

对于以上官方的处理 我后来采用的是多project+cacheDir+限制qb上传数的形式 最大限度减少对同一个project的api请求 我目前每个project日请求数在2万左右 一年多还算ok 但按目前Google的政策走势 只会越来越收紧 上半年提示之后可能会涨价 再加上前几天突然提示非组织账号无法创建文件 出于安全考虑 我已经把大部分资源拉回本地了 现在权当一个离线网盘使用 而且随着rclone支持pikpak之后 我考虑下载走pikpak是否可行

tyhunter commented 1 year ago

@dogbutcat 感测大佬的回复,我看了下您说的几点,自查了一下:

  1. 100个SA确实都是在1个project下面的,并且最近日均请求应该都在30w次以上,应该是两台机器同时在跑1个project的原因
  2. 项目目前还没被警告,就是只出现了403 userratelimit的限制,今天上午重新试了下恢复了,但是不确定会不会继续出现
  3. 想请教一下,多project+cacheDir+限制qb上传数,这个如何做到,是定期任务把下载完的文件全部传到GD吗?
tyhunter commented 1 year ago

@dogbutcat 我其实想的是有没有办法把普通google 账户当成server account来用?我大概也不需要这么多SA,每天有个2-3TB(大概7503-7505),就是3-5个普通账户来当成SA使用就可以了

dogbutcat commented 1 year ago

您说的情况看来是比之前松了 不清楚次数多了之后会不会有影响

多project+cacheDir+限制qb上传数

至于您说的把普通账户当成sa使用 在使用时两者的功能基本一致 具体见rclone的service-account-support以及google的what-are-service-account 而且OAuth2依赖client id和client secret 从维护角度讲更麻烦 实现意义不大

tyhunter commented 1 year ago

@dogbutcat 感谢大佬的耐心解答,我大概理解了 我再研究修改下上传脚本的逻辑,辛苦大佬