Reduce YouTube API quota usage

[calzoneman]

Background

• YouTube's Data API enforces quota for each API key
• The default quota was lowered to 10,000 units, and even keys grandfathered into the old limit had their limits adjusted (the current limit for cytu.be is 450K)
• Once the limit is exceeded, no requests to YouTube can succeed until the quota resets at midnight PST

Quota usage

Use Case	Units	Explanation
Single video queue	7	1 unit per API request, 2 units each for status,contentDetails,snippet
Playlist queue	402 per 50 item page	list playlist items (1 + 2*50) + lookup metadata for each item (1 + 6*50)
Search	251 per 25 item page	100 per search request + (1 per list request + (25 items per page)*(6 units per item))

Possible solutions/workarounds

• Request a quota increase, which requires filling out a long form with justifications -- I tried this last night and the form failed to submit for unknown reasons (and knowing YouTube it would be impossible to get any support so I think we can rule out this solution as infeasible)
• From a basic log analysis, it seems like we could save a decent amount by caching YouTube video responses with the returned ETag and doing a cache revalidation (still costs 1 unit to make the request, but saves 6 units if the video hasn't changed -- the particular 3 parts CyTube requests probably don't change often)
• Throttle/remove search -- at least remove the logic of defaulting search to YT if no library results are found and evaluate impact
• Throttle YouTube API requests to enforce the quota is used at a sustainable rate throughout the day instead of running out of quota at 9pm and taking a 3 hour outage of YouTube queues
• Load balance across multiple API keys? Given the default limit is 10K this may not help all that much, and I would need to check if it's against their ToS for the API
• Group videos in batches when making API requests -- probably won't save a lot for CyTube since the per-second single video lookup rate is generally pretty low (playlist lookups are already batched) and the cost of making a request is only 1/6th the cost of each video result

[calzoneman]

From 2/26 onward there is a steep dropoff in cache hit rate compared to found rate (implying a cached row exists but does not return a 304). Need to investigate what changed.

[calzoneman]

Cache hit rate recovered, but we hit the limit again. Next steps:

1. Add more monitoring around how much quota is spent on each type of request (single vid, playlist, search)
2. Add more logging around who is using up YouTube quota to track down any heavy hitters and evaluate how to smooth out traffic

[calzoneman]

Also need to fix the error message.

[dyc3]

I'm having the exact same problem on my project, OpenTogetherTube. I've used up all my quota for the past 4 days, mostly on search results. I'm considering using web scraping to do youtube searches instead. Here's some of my strategies to reduce quota usage:

• Heavy caching (which I can see you are already doing), and manual preemptive caching on days with low usage (although its not that effective)
• Rate limiting searches
• Only display 8 items for search results
• Only display 30 items for playlists
• Some parts don't need to be refreshed. Since my project only uses the video duration from contentDetails and that can't change, it won't request that again in the future.
• When we run out of quota, the site can still play videos, but we still need the video duration for the site to work so we scrape it from the webpage.

Where did you find the specific formulas for calculating quota usage? I can't find anything super useful on the official docs.

Also, using multiple API keys across different projects is against the youtube api ToS.

[calzoneman]

Are you still operating with the default limit of 10,000 units? If so, yeah, that will be spent pretty quickly. In ye olden days, the default was 50,000,000, into which my API key was grandfathered, but they recently adjusted that automatically and capped me at 450,000.

I submitted a request last night to ask them to bump me to 550,000 since I already was asked to complete a mandatory audit recently, so I already had most of the materials ready to fill out the form.

The calculations above are specific to the request pattern that I make for these scenarios, but I computed it by hand by adding up numbers from the docs, e.g. https://developers.google.com/youtube/v3/docs/videos/list#part and https://developers.google.com/youtube/v3/docs/search/list.

[calzoneman]

There's also this calculator, but I haven't tried it: https://developers.google.com/youtube/v3/determine_quota_cost

[dyc3]

Yeah I'm under the 10,000 limit. It sucks. I put in a request a few days ago for more quota, but I doubt I'll get a response anytime soon because of the coronavirus.

I found that calculator as well, but it's not super useful.

[calzoneman]

Update: for CyTube, most of the usage comes from the search functionality. I've temporarily disabled search and saw a drastic decrease in quota usage. Next options:

1. Limit the functionality of search in order to reduce the quota usage of each search (e.g. reducing quantity returned, avoiding needing to fetch additional fields)
2. Heavily rate limit search and enforce a strict quota limit on search
3. Remove search entirely (and instead have the button link to the YouTube search page)

[Xars1]

It probably needs to be examined; how many people actually use the search functionality on cytube. I for one have never use the search feature on cytube. Simply searching on separate tab has been sufficient. Frankly speaking, the search on cytube doesn’t anything to the platform, and doesn’t add anything to the search engines of youtube, vimeo, dailymotion, etc… In my opinion, removing the search functionality from cytube would most likely not be noticed by the vast majority of users.

[Xaekai]

@Settit: there is still channel libraries, which get used almost constantly, so the search box isn't going anywhere.

I still believe YT search box should be ran as a separate project, and thus ran under it's own API key.

But if it must be torn out, I suggest replacing it with a Vimeo search. Since it was only some YT terms very specific about inclusion in a mixed list of video search results being disallowed that prevented it's inclusion long ago.

[dyc3]

I just got a response to my API quota request, and apparently it's a ToS violation to store information from the youtube API for more than 30 days, and I need to link to Youtube's ToS in my ToS/privacy policy page.

Just giving y'all a heads up, you may get a similar response.

[calzoneman]

Yep I'm aware, YouTube audited my application before my quota increase request and my service terms were updated accordingly.

[Kirtaner]

YouTube demanded I remove several player-interacting controls like the skip video button and whatnot, then penalized me with a 50% cut in quota, etc. Whew.

[calzoneman]

Closing since CyTube is getting by with disabled search.