download quota exceeded

[jkoopmann]

Hi,

I am getting this:

[USR/LOCAL/BIN/PLEXDRIVE] [2018-02-04 18:03] WARNING: Could not read object XXX / StatusCode: 403 [USR/LOCAL/BIN/PLEXDRIVE] [2018-02-04 18:03] DEBUG : Starting download XXX (preload: false) [USR/LOCAL/BIN/PLEXDRIVE] [2018-02-04 18:03] DEBUG : { "error": { "errors": [ { "domain": "global", "reason": "downloadQuotaExceeded", "message": "The download quota for this file has been exceeded." } ], "code": 403, "message": "The download quota for this file has been exceeded." } }

I changed the naming of tons of files and plex is trying to get things sorted out so this might be related? Is this a coding issue or just me running into some GDRIVE limitation?

Regards, JP

[ddurdle]

You ran into an issue. You downloaded the file too many times (did you share it with others?).

To resolve it, in Google Drive web site, navigate to file, on the file, right-click, "make a copy". Delete the original file. Remove "copy of.." from the new filename. Issue should be resolved.

[jkoopmann]

I am not sharing. Happened to nearly all files and requests. I created new API credentials, reset plexdrive and so far it has not resurfaced. Keeping my fingers crossed.

[hrc1079]

Same thing has happened to me tonight. It spits out the error even on newly added files. First time I’ve seen this happen.

[ChrisColotti]

This just started for me too. Nothing will play and the API console shows hundreds of 403 errors with the. I have tried new API creds but it still comes back.

One thing I have is also a Plex Cloud instance connected to the same GDrive unlimited apps account hosting the files. So two servers connecting to the same location and neither are working. The only one I can see visible errors on is the VPS with PlexDrive running.

Last weekend I have accidentally share the folder (to myself) but I have since REMOVED the share but what I cannot tell is if google has permanently marked the files as shared even though they now are not. This is 8TB of data that I would have to copy, and you cannot in fact "copy" files in GDrive with the same user you basically would have to re-upload all of it from a backup. I'm not sure why after even I unshared it the files are still gacked after a week. the Ban on the files should have been removed by now

[silvether]

These issues are not related to PlexDrive. They are user related.

[hrc1079]

You offer no value by being so dismissive and I don't think they're not user related.

I have set up a new API key, rebuilt my entire system and it still keeps on happening. I did the initial sync yesterday and adding my libraries back into Plex and there are errors all over the place. My cache is 42GB and this has been happening for months now. I've made no changes to the system or command string

My library is still scanning and getting this all the time

[PLEXDRIVE] [2018-05-13 17:00] DEBUG : { "error": { "errors": [ { "domain": "global", "reason": "downloadQuotaExceeded", "message": "The download quota for this file has been exceeded." } ], "code": 403, "message": "The download quota for this file has been exceeded." } }

[PLEXDRIVE] [2018-05-13 17:00] DEBUG : { "error": { "errors": [ { "domain": "global", "reason": "downloadQuotaExceeded", "message": "The download quota for this file has been exceeded." } ], "code": 403, "message": "The download quota for this file has been exceeded." } }

[PLEXDRIVE] [2018-05-13 17:00] DEBUG : Checking for changes [PLEXDRIVE] [2018-05-13 17:00] DEBUG : Getting start page token from cache [PLEXDRIVE] [2018-05-13 17:00] DEBUG : Last change id found, continuing getting changes (287965) [PLEXDRIVE] [2018-05-13 17:00] DEBUG : Storing page token 287965 in cache

[dany20mh]

403 error and quota exceed it’s not necessary a PlexDrive issue and it’s pretty much depend on your usage.

Limits are 10TB download and 750GB upload, so you can hit those in any manner and it’s all depend on your setup and usage.

Making new API won’t help you that much, just check your usage and see what’s eating your quota.

[hrc1079]

It is common to have almost 450,000 API requests in 24 hours though?

[Requests] [Error] [Ratio] Google Drive API | 432,688 | 392,397 | 90.69%

The server has been unusable while I was syncing the media folders to Plex in the last 24 hours so I don't think it's a usage issue

Quotas

time	default
4/14/2018 0:00	68447
4/15/2018 0:00	66413
4/16/2018 0:00	51638
4/17/2018 0:00	49116
4/18/2018 0:00	54978
4/19/2018 0:00	63479
4/20/2018 0:00	54784
4/21/2018 0:00	70596
4/22/2018 0:00	49489
4/23/2018 0:00	60840
4/24/2018 0:00	59656
4/25/2018 0:00	44564
4/26/2018 0:00	98772
4/27/2018 0:00	54499
4/28/2018 0:00	53095
4/29/2018 0:00	25134
4/30/2018 0:00	73930
5/1/2018 0:00	21344
5/2/2018 0:00	30099
5/3/2018 0:00	33642
5/4/2018 0:00	21807
5/5/2018 0:00	41425
5/6/2018 0:00	31758
5/7/2018 0:00	28120
5/8/2018 0:00	25439
5/9/2018 0:00	15482
5/10/2018 0:00	15848
5/11/2018 0:00	20236
5/12/2018 0:00	185075
5/13/2018 0:00	196196

[YipYup]

Something definitely changed for you on the 12th. When you renamed files, it's possible that Plex is reanalyzing the media -- doing thumbnail generation, bitrate, audio and video codec analysis, etc. This could easily put your API request rate through the roof. I wonder if they're throttling you due to exceeding the request per second per user & requests per 100 seconds per user limits.

Also, you know what? 392,397 of your 432,688 requests were answered with a 403. If you renamed the files, I wonder if you should just flush the cache completely on plexdrive and start a new? Then you won't waste API calls on bad content? How did you do the rename? Through plexdrive or outside of it? I'm not 100% sure where I'm going with this, but just thinking aloud and trying to help.

[hrc1079]

The requests yesterday were due to me re-installing my plex server and having to add the media back info my server. I have about 38tb of content. What’s interesting though is that I’ve been getting banned daily since february.

I haven’t renamed my files. I re-installed plexdrive yesterday and built the cache database then added the libraries. I thought the cached db was supposed to take most the hits to avoid this happening. Guessing i misunderstood how it worked.

[dany20mh]

You still have something else using your drive data, I have 200TB and no issue what so ever with PlexDrive.

Also check your Plex settings too.

[hrc1079]

The only other thing I’m using is rclone to push the files to gdrive which I set up a different APi for (clearly makes no difference though). That runs on a different server.

Plexdrive is only running on this server. I’ve changed the library to scan every hour for new content only. Apart from that it’s not an overly complex setup. This is the plexdrive command i've been using -

plexdrive mount -c /home/plex/local/.plexdrive -v 3 -o allow_other,read_only --chunk-size=30M /home/plex/.gcd

[PLEXDRIVE] [2018-05-14 04:31] DEBUG : verbosity : DEBUG [PLEXDRIVE] [2018-05-14 04:31] DEBUG : root-node-id : root [PLEXDRIVE] [2018-05-14 04:31] DEBUG : config : /home/plex/local/.plexdrive [PLEXDRIVE] [2018-05-14 04:31] DEBUG : cache-file : /home/plex/.plexdrive/cache.bolt [PLEXDRIVE] [2018-05-14 04:31] DEBUG : chunk-size : 30M [PLEXDRIVE] [2018-05-14 04:31] DEBUG : chunk-load-threads : 4 [PLEXDRIVE] [2018-05-14 04:31] DEBUG : chunk-check-threads : 4 [PLEXDRIVE] [2018-05-14 04:31] DEBUG : chunk-load-ahead : 7 [PLEXDRIVE] [2018-05-14 04:31] DEBUG : max-chunks : 16 [PLEXDRIVE] [2018-05-14 04:31] DEBUG : refresh-interval : 1m0s [PLEXDRIVE] [2018-05-14 04:31] DEBUG : fuse-options : allow_other,read_only [PLEXDRIVE] [2018-05-14 04:31] DEBUG : UID : 1000 [PLEXDRIVE] [2018-05-14 04:31] DEBUG : GID : 1000 [PLEXDRIVE] [2018-05-14 04:31] DEBUG : umask : ---------- [PLEXDRIVE] [2018-05-14 04:31] DEBUG : Opening cache connection [PLEXDRIVE] [2018-05-14 04:31] DEBUG : Authorizing against Google Drive API [PLEXDRIVE] [2018-05-14 04:31] DEBUG : Loading token from cache [PLEXDRIVE] [2018-05-14 04:31] INFO : Mounting path /home/plex/.gcd [PLEXDRIVE] [2018-05-14 04:31] DEBUG : Checking for changes [PLEXDRIVE] [2018-05-14 04:31] DEBUG : Getting start page token from cache [PLEXDRIVE] [2018-05-14 04:31] DEBUG : Last change id found, continuing getting changes (287998) [PLEXDRIVE] [2018-05-14 04:31] INFO : First cache build process started... [PLEXDRIVE] [2018-05-14 04:31] DEBUG : Getting root from API [PLEXDRIVE] [2018-05-14 04:31] INFO : Processed 1 items / deleted 0 items / updated 1 items [PLEXDRIVE] [2018-05-14 04:31] DEBUG : Storing page token 287999 in cache [PLEXDRIVE] [2018-05-14 04:31] INFO : First cache build process finished!

[ChrisColotti]

I found out that a backup client for Code42 / Crashplan was killing mine. I am using the new Google Drive Stream and while nothing is downloaded and synced with that tool, Code42 was not smart enough to EXCLUDE the drive stream folder. Therefore it was trying to backup the 8TB of data in my online media folder. It killed the API as with drive stream, it has to first open/download the file to backup which is the API call. Once I put an ignore rule for that folder name it stopped exceeding the limit. I would agree that you have something else accessing the drive folder as was my case.

[dany20mh]

@hrc1079

1. You don’t need readonly
2. Change your cache chunk to 10Mb, this amount you don’t download too much of a file to analyze the file.
3. And don’t scan your library every 1 hour, that cause too much pressure, even if it’s just searching for new things

[hrc1079]

Thanks @dany20mh

I've changed the setting based on your recommendations but have been blocked again

time	default
4/15/2018 0:00	66413
4/16/2018 0:00	51638
4/17/2018 0:00	49116
4/18/2018 0:00	54978
4/19/2018 0:00	63479
4/20/2018 0:00	54784
4/21/2018 0:00	70596
4/22/2018 0:00	49489
4/23/2018 0:00	60840
4/24/2018 0:00	59656
4/25/2018 0:00	44564
4/26/2018 0:00	98772
4/27/2018 0:00	54499
4/28/2018 0:00	53095
4/29/2018 0:00	25134
4/30/2018 0:00	73930
5/1/2018 0:00	21344
5/2/2018 0:00	30099
5/3/2018 0:00	33642
5/4/2018 0:00	21807
5/5/2018 0:00	41425
5/6/2018 0:00	31758
5/7/2018 0:00	28120
5/8/2018 0:00	25439
5/9/2018 0:00	15482
5/10/2018 0:00	15848
5/11/2018 0:00	20236
5/12/2018 0:00	185075
5/13/2018 0:00	296042
5/14/2018 0:00	14662

The requests have gone back down to normal since re-adding the library to Plex so hopefully it settles down again and prevents blocking. I've looked through everything to see if something else is accessing Google Drive and it's definitely only Plexdrive & Rclone.

[ChrisColotti]

I also had an issue where I shared my media folder to myself (another Google user account) and was accessing that via PLexDrive and I was getting blocked. I had to unshare the folder and access the original file via the folder owner directly. I was never exceeding the API quotas either but they blocked it due to the fact share access is tracked differently. You can only access so much data a data from a shared folder. Once I unshared it and connected directly to the folder as the owner, no issues except that backup utility and Drive Stream. It took about 48 hours for all the locks to clear out after it was unshared and I could maintain a normal level of connectivity. I currently average 4-6 concurrent streams on my VPS with Plex Drive (only 8TB library) with no issues.

[YipYup]

@ChrisColotti Was that with a personal drive that was shared, or was that with a team drive?

[ChrisColotti]

It was a google Apps "My Drive" (not a team drive) but I shared it to my Personal GMail account which I then connected Plex Drive to, and it did in fact get blocked due to different rules on sharing. Anything you "share" with Google gets lumped into some other lock out policy and I was getting the exact same "download error". If you are accessing it via an original folder shared, you WILL have issues. don't share your media just access it directly. That being said PlexDrive does not see Team Drives only your "My Drive" folders. Sharing a folder somehow changes the game I think I found some references to it online.

[felixbuenemann]

You could create server side copies of the shared files using either rclone or the google drive website and then playback from the copies.

[hrc1079]

I got banned again last night. If i'm reading the API section correctly in Google this is what I've got -

Plexdrive API 1.29 requests per second and 111,000 total over 1 day rClone API 0.063 requests per second and 5,441 total over 1 day

These are within the limits of the quotas so I've really got no idea where to even continue looking at this point. I'd open a case with Google but not sure if they would question seeing 30+TB of encrypted content and wonder what's going on.

**UPDATE - I think I may have found the issue. I'm using rClone mount on my downloading server. Some reports suggest that this is what may be causing the ban. I don't know why it just started occurring in February considering it had been running perfectly fine with that set up for a good 6 to 8 months beforehand. Going to change how I mount the downloading server into Google

[dany20mh]

@hrc1079

As I just mentioned before, it's not always about API, your download usage is hit the max and that's why you are getting banned. As for rclone, that will cause ban too, even if you use the cache mode.

[ChrisColotti]

as @dany20mh has said and as I saw myself i was blocked with the shared download limits. If your folder is shared at all...you will hit these download limits which are not part of the API limits. All your issues (speaking from personal experience) are user induced, not API or plexdrive related. I suspect your media folder is shared and needs to be unshared completely.

[hrc1079]

Yeah, after getting more detail on the API requests yesterday it's definitely not Plexdrive. I've turned off my server running rClone and haven't been blocked as yet. Guess I need to sort out a way to sync the downloaded content to Google differently.

Definitely can confirm that none of the folders are shared from my google drive.

[dany20mh]

I’m not sure what’s your setting with rclone sync, because I’m using it and I don’t have issue and it’s working pretty fine

[hrc1079]

/usr/sbin/rclone copy /home/plex/.local/(folder)/ google:/(folder) &$

This worked fine for months. Only started really acting up since February

[dany20mh]

It’s because you just don’t listen, google drive have upload and download limit, and you keep hitting it with that command.

[hrc1079]

I’m not uploading 750Gb or Downloading 10Tb a day!

[felixbuenemann]

Btw. older rclone versions often needed --no-traverse to avoid hitting api call limits during sync. Apparently this was fixed in recent versions, because the switch is now deprecated.

[JJohnson931]

Im also having the same issue as jkoopmann and hrc1079, however im not rebuilding the library or plexdrive's cache, but for the past 1-2 weeks ive been hitting some quota almost daily. The library and cache has already been built and fine for months, i switched to plex guide 2 or 3 months ago but imported plex's library, PD did rebuild its cache then, but without issues and has been stable and fine up until now

in the admin audit log ill see things like this:

go on and on and according to the history on tautili nobody is watching that file, other times ill see these:

which seem much more normal, and were files that were actually watched.

im using plexguide with PD 5 installed

Im NOT hitting the api limits (set for like 1,000,000,000 daily and 10,000 per 100 seconds and 10,000 i requested increased api limits) NOR am i downloading anywhere close to 10TB/day or uploading 750GB/day.

Also when the ban happens, it only affects video files, if, say, a random image file or text file is even in the same folder with an episode/movie file it will dowload those files just fine during the ban.

Also i thought all bans reset at midnight/PT since ive been having issues the bans have been lifting and weird/random times during the day

[ddurdle]

Bans don't revolve around a time. They work on what have you used in the past 24 hours at this time. so let's say you suddenly went crazy at noon and started downloading 10TB between noon and 3pm, and hit the cap at 3pm. At any given time between 3pm and noon the following day, your answer to how much quota data you've used in the past 24 hours would be 100%. But then at noon, that % will start dropping. It is hard to explain because it is such common sense.

[felixbuenemann]

@cm0002 Maybe you have video preview thumbnail generation activated in your Plex server? That would cause a lot of download requests.

To a lesser degree that also applies to chapter thumbnail generation, though I have that activated on my server without problems.

[JJohnson931]

@felixbuenemann I was pretty sure i disabled it, but I double checked just to make sure and it was definitely disabled already from the beginning