search

int128 / gpup

A command to upload photos and movies to Google Photos Library using the official Google Photos Library API
Apache License 2.0
220 stars 22 forks source link

Hitting uploads per minute quota #13

Closed jone closed 2 years ago

jone commented 5 years ago

Hey there

When uploading large amounts of photos at once I'm hitting a quota:

2018/09/15 19:17:26 Uploading IMG_6427.jpg
2018/09/15 19:17:26 Error while uploading /Users/jone/Pictures/Photos Library.photoslibrary/Masters/2009/07/18/20090718-190427/IMG_6663.jpg: Could not upload /Users/jone/Pictures/Photos Library.photoslibrary/Masters/2009/07/18/20090718-190427/IMG_6663.jpg: status 400: {
  "code": 3,
  "message": "Insufficient tokens for quota \u0027photoslibrary.googleapis.com/upload_requests\u0027 and limit \u0027UploadsPerMinute\u0027 of service \u0027photoslibrary.googleapis.com\u0027 for consumer \u0027project_number:638493XXXXX\u0027."
}

Unfortunately I'm not that familiar with Go, but as far as I could interpret the code I suspect that gpup does not detect that problem and retry uploading this particular image: the response status is 400, but retry does only work with codes of 500 and greater. I guess there is no way to circumvent the quota, with another type of account, right? Or did I do something wrong?

I'd suggest to implement a sleep / retry strategy, but I'm not familiar enough with the Go environment to do that on may own.

pacoorozco commented 5 years ago

Hi @jone

This is the rate limit that Google is applying to Google Photos API, so It's difficult to avoid this issue.

General quota limit The quota limit for requests to the Library API is 10,000 requests per project per day. Requests to access media bytes (like loading a photo or video) are not counted against the limit.

Exceeding quota limits If the quota of requests to the Library API is exceeded, the API returns an error code 429 and a message that the project has exceeded the quota. For more information, see the Terms of Service.

felciano commented 5 years ago

@pacoorozco or @int128, is there a straightforward way to restart an upload sequence that failed due to this quota error? Or is there a way to get gpup to avoid uploading media that was previously successfully uploaded (e.g. via checksum comparison btw local media file and the remote one)?

pacoorozco commented 5 years ago

Hi @felciano

I'm using this script that keeps track of uploaded directories, so I can resume it later or I can do periodic syncs. You are free to modify it to keep track of files (instead of directories). Bear in mind that Google doesn't allow to upload twice the same file :-D

Feedback is welcomed :-D

endzyme commented 4 years ago

https://github.com/int128/gpup/pull/46 (modifying the support for more retry-ability could be done like so). Either with status code or with status code and error message.