Does not respect/aware of google quota for request limit

[homiak]

I've been using gphotos-uploader-cli for a past couple of months and uploaded quite a few photos, videos to Google Photos Lately I've noticed the upload would stack for a very long time after this step

[info] Found 3349 items to be uploaded processing location '/Volumes/DATA/PHOTOS'.

And then it would start giving

[fail] X Unable to create album 'portraits_Feb2014': Post "https://photoslibrary.googleapis.com/v1/albums?alt=json": POST https://photoslibrary.googleapis.com/v1/albums?alt=json giving up after 5 attempt(s)

I checked the google cloud console and see that I've reached the quota limits with most of the requests spent on google.photos.library.v1.PhotosLibrary.ListAlbums

I see two problems here First and most obvious is that gphotos-uploader-cli does not respect HTTP 429 replies and keep retrying.

Second is more complex - if I have 10000 albums, and the default Google quota of 10000 requests per day, does it mean I can no longer user gphotos-uploader-cli, simply because to do anything it would have to list all the albums first and run out of the requests? Please note that I configured gphotos-uploader-cli to look into the new folder where none of the albums have been previously uploaded, and there are maybe only a 100 albums. To me it should not be making 10000 requests in this situation. Of course there is a local database that is likely to contain a lot of albums (all previously uploaded), but that should not make an impact.

I am using 3.1.0 (but same issue affected the previous version)

Might be related to https://github.com/gphotosuploader/gphotos-uploader-cli/issues/293

[homiak]

[homiak]

[homiak]

[homiak]

Just checked - I do not have 10000 albums, I have less than 1000. From what I see the problem might be related to the fact that I have changed the configuration for CreateAlbums mid way from folderName to folderPath.

[pacoorozco]

Hi @homiak

There's something that I'm not able to understand...

[info] Found 3349 items to be uploaded processing location '/Volumes/DATA/PHOTOS'.

If you have 3349 items you should not be able to reach the 10.000 reqs limit.

gphotos-uploader-cli does a lot of things to reduce the number of requests:

1. Group items by albumName.

You can see here, how all the items are grouped by albumName (independently if you use folderName or folderPath).

2. Cache is used when calling Google Photos

To get the albumId uses a client which is caching all requests, that reduces the number of API calls, see it here.

3. Uses one call for each file

It uses only one call to upload the file, given that the albumId was provided before, here

On my lab I've seen how the gphotos-uploader-cli was able to upload more than 6000 files before reaching the limit. That's what is concerning me, because you have reach the same point after less than 3400 files 🤷🏼‍♂️

There is an idea in the backlog, which is to populate the cache during startup (it has not implemented yet), that will reduce the number of requests when listing albums drastically.

For your specific problem. I'll suggest to upload your files in smaller tasks. Using the folders that you have inside '/Volumes/DATA/PHOTOS'. Note that you should maintain the same source folder and change the IncludePatterns or ExcludePatterns.

[homiak]

For your specific problem. I'll suggest to upload your files in smaller tasks. Using the folders that you have inside '/Volumes/DATA/PHOTOS'. Note that you should maintain the same source folder and change the IncludePatterns or ExcludePatterns.

Thanks for your support Paco. In the past I have already changed the source folder a couple of times. Worked fine, but I am not sure what deeper effects that could have had.

I am planning to run gphotos-cli one more time tomorrow to see if it works. If not I am planning to discard the internal db for gphotos-cli something is telling me it is causing the issue. I will let you know how it went.

[pacoorozco]

The main difference of changing or not SourceFolder is how ghpotos-uploader-cli will track already uploaded files. It uses the relative path (from SourceFolder) so It's preferable to not modify it and apply IncludePatterns to choose which files should be uploaded.

[homiak]

I am please to report that my problem has been resolved, and it seem be have been unrelated to gphotos-uploader-cli. I tried on Monday and ended up with the same result - gphotos-uploader-cli made 10000 requests to ListAlbums and got blocked. (at least according to Google dashboard) In fact gphotos-uploader-cli was completely blocked on the remote call. I would suggest using timeouts, but that is a separate story anyway. I did not touch anything for a few days, apart from deleting a few albums and uploading pictures to google photos via the web browser. I do not think it did change anything, but maybe Google fixed something on their end. At least today I give gphotos-uploader-cli another chance, and it started working just as before. I can see in google dashboard the sane number of API calls (as much as it would have been necessary). So yeah, it's working. Still it would be nice to address the issue or not been able to distinguish the Quota limit replies, as well as use timeouts to avoid hanging when the quota is reached.

[homiak]

Update: after completing pushing one of my source folder to G photos successfully, I pointed gphotos-uploader-cli to another folder (via config) with about 4.5K files and a few dozens of albums. I had the same problem come back. gphotos-uploader-cli would consume all Google Photos quota mainly by ListAlbums calls. I thought this might be because I have uploaded quite a few files already or maybe a bug in caching. So I re-initialized gphotos-uploader-cli completely by remove previous config directory and running init. Moreover I created a new project in google and new credential set for it (I was already over my quota in previous projects). So with new setup where gphotos-uploader-cli would not now anything about what has already been uploaded to Gphotos and no local database (excluding any potential caching issues) I run it again. To my surprise, it started to upload a few files and created a few albums but the upload was very slow. I checked the API usage stats and saw that we started to rake listAlbums calls again. When it reached 1000 requests I terminated it, and hooked a debugger to it. That confirmed that we got stuck paging in func (c *AlbumsListCall) Pages(ctx context.Context, f func(*ListAlbumsResponse) error) error So looks like a concurrency issue to me, or something with result pagination. See below for the full stack trace. If you fancy a zoom call we can do it, and if we are lucky we might be able to reproduce the issue.

[homiak]

[homiak]

[homiak]

I am sorry for spamming this issue - it's no longer related to quota for request limit but I trust you will figure it out.

[Tantalon]

I'm experiencing the same problem - ListAlbums calls (far in excess of the # of albums) are exceeding the quota limit