search

xyou365 / AutoRclone

AutoRclone: rclone copy/move/sync (automatically) with thousands of service accounts
https://www.gfan.loan/?p=235
1.36k stars 516 forks source link

Getting error 403 rate limited after short time copying #71

Open RyanWor opened 3 years ago

RyanWor commented 3 years ago

I believe autorclone is triggering the "Queries per 100 seconds per user" quota in Drive which is 10,000 on my GSuite account. THis means my transfer works great and is going very fast until 3-3.5 TB and then hits API ban and stops copying data. Any idea how to prevent this from happening?

graphixillusion commented 3 years ago

@RyanWor Same is happening to me. Did you find any solution?

graphixillusion commented 3 years ago

Any news about this?

poornachandratejasvi commented 3 years ago

even im facing same issue.. any solution plz

RyanWor commented 3 years ago

If you are having the same issue as I was it is likely because you are hitting the third undocumented Google Drive limit. This limit was introduced in the last year or so I think and limits server side copies to a max per owner per day. I think this is somewhere around 2-3TB but not certain the exact number. More discussion on it at the rclone forum link below. If you are trying to copy a large amount of data from a My Drive to a Team Drive all that source data has a single owner (the account that uploaded the data to it's own personal My Drive). If you are in that boat unfortunately you must first migrate all that data into a Team Drive using service accounts as if it were a local upload. I choose to mount my My Drive remote and then use the "For local to Google Drive" method at the bottom of the readme to upload that data into my Team Drive. There may be other ways to achieve this as well, not sure. Using this method the files have numerous different owners (could by any of your service accounts) and then future Team Drive to Team Drive copies shouldn't really suffer this issue.

https://forum.rclone.org/t/google-drive-is-nerfing-service-accounts/19049