Benjamin-Loison
commented
9 months ago It seems that a potentially not considered YouTube Data API v3 error that I faced may break the no-key endpoint. Just a reminder of the well-known YouTube Data API v3 quota exceeded error:
{
"error": {
"code": 403,
"message": "The request cannot be completed because you have exceeded your <a href=\"/youtube/v3/getting-started#quota\">quota</a>.",
"errors": [
{
"message": "The request cannot be completed because you have exceeded your <a href=\"/youtube/v3/getting-started#quota\">quota</a>.",
"domain": "youtube.quota",
"reason": "quotaExceeded"
}
]
}
}https://developers.google.com/youtube/v3/docs/errors https://developers.google.com/youtube/v3/docs/core_errors
It seems that using 403 instead of the actual reason would treat both cases.
At least some other temporary error that I sometimes faced would not be a problem as it is 503: https://developers.google.com/youtube/v3/docs/core_errors#SERVICE_UNAVAILABLE
Could maybe be more precise among the 403 errors, as some are temporary in theory, like: concurrentLimitExceeded, dailyLimitExceeded (should verify not being in the case The daily quota limit has been reached, and the project has been blocked due to abuse. See the [Google APIs compliance support form](http://support.google.com/code/go/developer_compliance) to help resolve the issue.), rateLimitExceeded, servingLimitExceeded, userRateLimitExceeded, variableTermExpiredDailyExceeded and variableTermLimitExceeded.
Should check if can put 0 limit for above limits and if so, then should verify that the key worked once before adding.
As most people do not use such features and assume that the no-key endpoint is not too much used to trigger a specific one, let us drop at least temporarily forbidden keys, i.e. other than quota exceeded 403 keys.
limitExceeded seems ambiguous.
dailyLimitExceededUnreg, rateLimitExceededUnreg and userRateLimitExceededUnreg seem to require signing.
Also just faced:
{
"error": {
"code": 400,
"message": "API key expired. Please renew the API key.",
"errors": [
{
"message": "API key expired. Please renew the API key.",
"domain": "global",
"reason": "badRequest"
}
],
"status": "INVALID_ARGUMENT",
"details": [
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": "API_KEY_INVALID",
"domain": "googleapis.com",
"metadata": {
"service": "youtube.googleapis.com"
}
}
]
}
}Faced on https://yt.lemnoslife.com OP error and afterwards had less than expected YouTube Data API v3 keys it seems as a significant part was missing:
wc -l ytPrivate/keys.txt1023 /var/www/yt/ytPrivate/keys.txtsed -i '/^\s*$/d' ytPrivate/keys.txt
wc -l /var/www/yt/ytPrivate/keys.txt1023 /var/www/yt/ytPrivate/keys.txtpython3 test.pycorrect=0 quotaExceeded=1024: 100%|█| 1024/1024 [01:39<0Afterwards got:
./test.pycorrect=2319 quotaExceeded=1306 toRemove=2: 100%|██████████████████████████████████████████████████████████████████████████| 3627/3627 [08:15<00:00, 7.32it/s]#!/usr/bin/python3
import requests
import json
from tqdm import tqdm
keysFilePath = 'ytPrivate/keys.txt'
with open(keysFilePath) as f:
lines = f.read().splitlines()
def getValue(obj, path):
if path == '':
return obj
pathParts = path.split('/')
key = pathParts[0]
key = int(key) if key.isdigit() else key
try:
newObj = obj[key]
return getValue(newObj, '/'.join(pathParts[1:]))
except:
return None
correctKeys = set()
quotaExceededKeys = set()
toRemove = 0
progressBar = tqdm(lines)
for line in progressBar:
url = 'https://www.googleapis.com/youtube/v3/videos'
params = {
'part': 'snippet',
'id': '_ZPpU7774DQ',
'key': line,
}
data = requests.get(url, params = params).json()
if 'error' in data:
error = data['error']
if error['errors'][0]['domain'] != 'youtube.quota':
message = error['message']
if message == 'API key expired. Please renew the API key.' or message.startswith('has been suspended.') or message == 'API key not valid. Please pass a valid API key.' or message == 'API Key not found. Please pass a valid API key.' or message.startswith('YouTube Data API v3 has not been used in project ') or message.endswith('are blocked.'):
toRemove += 1
else:
print(line)
print(json.dumps(data, indent = 4))
break
else:
quotaExceededKeys.add(line)
elif getValue(data, 'items/0/snippet/channelId') == 'UCWeg2Pkate69NFdBeuRFTAw':
correctKeys.add(line)
else:
print(line)
print(json.dumps(data, indent = 4))
break
progressBar.set_description(f'{len(correctKeys)=} {len(quotaExceededKeys)=} {toRemove=}')
# Manually sort keys like PHP script would also be nice but then use the PHP script.
# In theory could better sort keys based on original order, maybe current implementation keep it?
with open(keysFilePath, 'w') as f:
f.write('\n'.join(list(correctKeys) + list(quotaExceededKeys)))Should make the no-key endpoint use a function and make a PHP script based on this function to make sure that it is the production no-key endpoint behavior. Could use the Stack Overflow answer 27147177 to have a progress bar.
Hence, solved this issue thanks to uploading the complete key set.
This time the 5th line was empty on the official instance.
I believe that this issue comes from the fact that the web server treats multiple requests at the same time. Could for instance introduce a blocking mutex on the file read and write but avoiding file read would be nice thanks to across requests memory.
Related to #237 and more precisely this comment.