RateLimited during url inspection

[maximepvrt]

Currently, when conducting an inspection for indexing using the tool, I encountered an obstacle. It appears that there is a rate limit restriction of 2000 calls per day (https://support.google.com/webmasters/thread/240916045/429-quota-issue-googleapis-com-v1-urlinspection-index-inspect?hl=en), which significantly hampers the inspection process.

👍 Done, here's the status of all 14784 pages:
🚦 RateLimited: 14784 pages

{
  "error": {
    "code": 429,
    "message": "Quota exceeded for sc-domain:xxxxxxxxx.fr.",
    "status": "RESOURCE_EXHAUSTED"
  }
}

Proposed Enhancement

• Cache-Based Inspection Calls: Introduce use the caching mechanism to use previous inspection results and only inspect URL with 429 Responses.

• Rate Limit Management: Implement a rate limit strategy for inspection calls to ensure adherence to the daily limit of 2000 calls. This could involve throttling the rate of inspection requests to stay within the allowed quota.

• Error Handling for 429 Responses: Identify and block inspection requests for URLs that were not analyzed due to encountering a 429 error (rate limit exceeded). This prevents redundant calls for URLs that are already queued for inspection.

• Bypass Mechanism for Cached URLs: Introduce a mechanism to bypass inspection for URLs already present in the cache. This would allow for direct indexing of URLs stored in the cache, optimizing the indexing process.

[lundcm]

I have the following locally as a bit of a hack but it works for my purposes. This is in utils.ts:

export async function fetchRetry(url: string, options: RequestInit, retries: number = 5) {
  try {
    const response = await fetch(url, options);
    if (response.status >= 500) {
      const body = await response.text();
      throw new Error(`Server error code ${response.status}\n${body}`);
    }

    if (response.status === 429) {
      console.log("Rate limited. Retrying in 5 seconds...");
      // Retry after 5 seconds if rate limited
      await new Promise((resolve) => setTimeout(resolve, 5000));

      // Retry the request ignoring the retries limit
      return fetchRetry(url, options, retries);
    }
    return response;
  } catch (err) {
    if (retries <= 0) {
      throw err;
    }
    return fetchRetry(url, options, retries - 1);
  }
}

[muditjuneja]

This works like a charm for me!! Thanks @lundcm