`Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.` with the no-key endpoint

[Benjamin-Loison]

{
  "error": {
    "code": 403,
    "message": "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.",
    "errors": [
      {
        "message": "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.",
        "domain": "global",
        "reason": "forbidden"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

Related to #229 (hence also this initial Git issue comment).

[Benjamin-Loison]

#!/usr/bin/python3

import requests
import json
from tqdm import tqdm

with open('ytPrivate/keys.txt') as f:
    lines = f.read().splitlines()

def getValue(obj, path):
    if path == '':
        return obj
    pathParts = path.split('/')
    key = pathParts[0]
    key = int(key) if key.isdigit() else key
    try:
        newObj = obj[key]
        return getValue(newObj, '/'.join(pathParts[1:]))
    except:
        return None

correct = 0
quotaExceeded = 0
progressBar = tqdm(lines)
for line in progressBar:
    url = 'https://www.googleapis.com/youtube/v3/videos'
    params = {
        'part': 'snippet',
        'id': '_ZPpU7774DQ',
        'key': line,
    }
    data = requests.get(url, params = params).json()
    if getValue(data, 'items/0/snippet/channelId') == 'UCWeg2Pkate69NFdBeuRFTAw':
        correct += 1
    elif data['error']['errors'][0]['reason'] == 'quotaExceeded':
        quotaExceeded += 1
    else:
        print(json.dumps(data, indent = 4))
        break
    progressBar.set_description(f'{correct=} {quotaExceeded=}')

with https://yt.lemnoslife.com key set shows that no key still have any quota.

Note that I ran this script in 18 seconds on my Linux Mint Framework and I tested manually a few keys in Firefox and noticed only quota being exceeded.

Are all keys somehow empty, if they can even be or there was more quota usage than usually? I should monitor key usage.

In the meanwhile I should look for new keys.

I communicated on Discord concerning this main issue.

Above script helped me figure this no more interesting key:

{
    "error": {
        "code": 403,
        "message": "Permission denied: Consumer 'api_key:AIzaSyB2XeDcarYSa-TFY1ga_lTrlk9mOzVXIQM' has been suspended.",
        "errors": [
            {
                "message": "Permission denied: Consumer 'api_key:AIzaSyB2XeDcarYSa-TFY1ga_lTrlk9mOzVXIQM' has been suspended.",
                "domain": "global",
                "reason": "forbidden"
            }
        ],
        "status": "PERMISSION_DENIED",
        "details": [
            {
                "@type": "type.googleapis.com/google.rpc.ErrorInfo",
                "reason": "CONSUMER_SUSPENDED",
                "domain": "googleapis.com",
                "metadata": {
                    "service": "youtube",
                    "consumer": "projects/137880112907"
                }
            }
        ]
    }
}

Once have treated this issue should unpin it and set notifyOnError back to True in /var/www/ytPrivate/checkQuota.py on LemnosLife VPS. Notify private instance owners in addition to send a message to #announcements Discord channel.

On https://yt.lemnoslife.com:

grep -n -m 1 '13/Feb/2024:00:56' access.log.1

15260612:CENSORED

wc -l access.log access.log.1

    9201889 access.log
   15960445 access.log.1
   25162334 total

Hence 15,960,445 - 15,260,612 + 9,201,889 = 9,901,722 have been performed in the last 24 hours.

Communication concerning this big number on Discord.

[Benjamin-Loison]

As below script run under 16 seconds on OC3K, I programmed a crontab to run every minute to let me know once a key has quota.

#!/usr/bin/python3

import requests
import json

with open('/var/www/yt/ytPrivate/keys.txt') as f:
    lines = f.read().splitlines()

for line in lines:
    url = 'https://www.googleapis.com/youtube/v3/videos'
    params = {
        'part': 'snippet',
        'id': '_ZPpU7774DQ',
        'key': line,
    }
    data = requests.get(url, params = params).json()
    if data['error']['errors'][0]['reason'] != 'quotaExceeded':
        print(json.dumps(data, indent = 4))
        break

[Benjamin-Loison]

Solving #91 would make the no-key endpoint and its maintenance no more important.

[Benjamin-Loison]

On https://yt0.lemnoslife.com I noticed in /var/log/apache2/ after a reboot:

==> error.log <==
[Thu Feb 15 23:27:59.543673 2024] [mpm_prefork:error] [pid 1023] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Maybe this Stack Overflow answer helped making https://yt0.lemnoslife.com answer request instead of timing out.

yt.lemnoslife.com-ssl--access.log seems to show many requests to its no-key endpoint for https://yt0.lemnoslife.com.

[Benjamin-Loison]

Manually merging keys of:

• Linux Mint Framework
• https://yt.lemnoslife.com
• https://yt0.lemnoslife.com
• https://yt4.lemnoslife.com
• private instances

Verified exhaustiveness thanks to LemnosLife VPS /etc/bind/db.lemnoslife.com.

Search keys in logs:

• https://yt.lemnoslife.com
• https://yt0.lemnoslife.com
• https://yt4.lemnoslife.com
• private instances

grep -Poh 'AIzaSy[A-D][a-zA-Z0-9-_]{32}' | sort | uniq -u
(cat access.log{,.1} && gunzip -c access.log.*.gz) | grep -Poh 'AIzaSy[A-D][a-zA-Z0-9-_]{32}' | sort | uniq -u

[Benjamin-Loison]

Consider #18.

[Benjamin-Loison]

Unclear what was current assumption but let us now keep all keys even those that are quota exceeded, assuming that no key can have no quota forever.

[Benjamin-Loison]

With new keys.txt:

correct=2056 quotaExceeded=1571: 100%|█████████████████████████████████████████████████████████████████████████████████████| 3627/3627 [05:56<00:00, 10.19it/s]

Previously had about 268 keys.

May have issues due to file reading with so many keys.

[Benjamin-Loison]

TLDR: incident solved (for the moment at least), you can use the no-key endpoint as previously Previously the no-key endpoint was empowered by about 262 YouTube Data API v3 keys that I mostly found by hand on the Internet. However, as I said in the last hours all keys were having their quota exceeded. Hence, I wrote web-scraping algorithms to automatize my Internet search for YouTube Data API v3 keys and I found a very efficient source which resulted in 3,627 keys with 2,056 not currently having their quota expired. I updated the official and private instances key set with the one I elaborated. Except if Google is scrutinizing my work, we should no more have any quota related issue concerning the no-key endpoint for a long time.

[Benjamin-Loison]

Note that for unknown reason https://yt0.lemnoslife.com no-key endpoint is not working but as it is not officially supported let us do not care about it for the moment.

[Benjamin-Loison]

Related to Benjamin-Loison/matrix-commander/issues/8.

[Benjamin-Loison]

Another possibility to get YouTube Data API v3 could be to get those of big companies, thanks to this method.

[Benjamin-Loison]

Just got this error, the second line of keys.txt was empty (or whitespaces only), I solved the issue by removing this empty line thanks to a Linux command. I forgot to apply this fix on private instances as well. Opened the dedicated issue #240.

[Benjamin-Loison]

import requests
from tqdm import tqdm

ytUrl = 'https://www.googleapis.com/youtube/v3/videos'
ytParams = {
    'part': 'snippet',
    'id': '_ZPpU7774DQ',
}

correctKeys = []
quotaExceededKeys = []

def getValue(obj, path):
    if path == '':
        return obj
    pathParts = path.split('/')
    key = pathParts[0]
    key = int(key) if key.isdigit() else key
    try:
        newObj = obj[key]
        return getValue(newObj, '/'.join(pathParts[1:]))
    except:
        return None

with open('b') as f:
    keys = f.read().splitlines()
    for key in tqdm(keys):
        ytParams['key'] = key
        data = requests.get(ytUrl, params = ytParams).json()
        if getValue(data, 'items/0/snippet/channelId') == 'UCWeg2Pkate69NFdBeuRFTAw':
            correctKeys += [key]
            print(f'{len(correctKeys)=}')
        elif data['error']['errors'][0]['reason'] == 'quotaExceeded':
            quotaExceededKeys += [key]
            print(f'{len(quotaExceededKeys)=}')

print('\n'.join(quotaExceededKeys + correctKeys))