Fix Github API rate limits

samajammin commented 3 years ago

Describe the bug

Our app queries the Github API quite aggressively, e.g. for

Commit histories for every docs / tutorial file
Project metadata (like stars)

The sheer number of requests we make to the Github API on site build (using gatsby-source-graphql) is frequently hitting rate limits, which often results in build failures, e.g. #2159:

9:59:35 AM: success run static queries - 1.990s - 9/9 4.52/s
10:03:51 AM: error There was an error in your GraphQL query:
10:03:51 AM: API rate limit exceeded
10:03:51 AM: File path: /opt/build/repo/src/templates/tutorial.js
10:03:51 AM: Url path: /fa/developers/tutorials/run-light-node-geth/
10:03:51 AM: Plugin: none

See full deploy logs.

To Reproduce See build errors above.

Expected behavior Look into ways to prevent hitting API rate limits

Ways we can cache results when developing locally?
Ways to prevent queries when developing locally (e.g. QUERY_ON_DEMAND flag - #2151)?
Use different API keys for developing locally vs. preview deploy vs. production?
Investigate batching requests, specifically apollo-link, which the Gatsby team recommended

Additional context

See this discussion on additional options: https://github.com/gatsbyjs/gatsby/discussions/28680

samajammin commented 3 years ago

Different error but appears to be this same cause:

8:31:16 PM: error There was an error in your GraphQL query:
8:31:16 PM: Response not successful: Received status code 403
8:31:16 PM:   12 |       title
8:31:16 PM:   13 |       description
8:31:16 PM:   14 |       incomplete
8:31:16 PM:   15 |       sidebar
8:31:16 PM:   16 |       sidebarDepth
8:31:16 PM:   17 |     }
8:31:16 PM:   18 |     body
8:31:16 PM:   19 |     tableOfContents
8:31:16 PM:   20 |   }
8:31:16 PM:   21 |   gitData: github {
8:31:16 PM: > 22 |     repository(name: "ethereum-org-website", owner: "ethereum") {
8:31:16 PM:      |     ^
8:31:16 PM:   23 |       ref(qualifiedName: "master") {
8:31:16 PM:   24 |         target {
8:31:16 PM:   25 |           __typename
8:31:16 PM:   26 |           ... on GitHub_Commit {
8:31:16 PM:   27 |             history(path: $relativePath) {
8:31:16 PM:   28 |               edges {
8:31:16 PM:   29 |                 node {
8:31:16 PM:   30 |                   message
8:31:16 PM:   31 |                   commitUrl
8:31:16 PM:   32 |                   author {
8:31:16 PM: File path: /opt/build/repo/src/templates/docs.js
8:31:16 PM: Url path: /hu/developers/docs/intro-to-ethereum/
8:31:16 PM: Plugin: none
8:31:16 PM: not finished Generating image thumbnails - 114.237s
8:31:16 PM: not finished run page queries - 113.996s
8:31:16 PM: error Command failed with exit code 1.

Here's the full build logs.

samajammin commented 3 years ago

Different error but appears to be this same cause:

5:14:05 AM: error "gatsby-source-graphql" threw an error while running the sourceNodes lifecycle:
5:14:05 AM: not finished source and transform nodes - 67.399s
5:14:05 AM: not finished Generating image thumbnails - 42.351s
5:14:05 AM: (sharp:1924): GLib-CRITICAL **: 13:14:05.217: g_hash_table_lookup: assertion 'hash_table != NULL' failed
5:14:05 AM: (sharp:1924): GLib-CRITICAL **: 13:14:05.230: g_hash_table_lookup: assertion 'hash_table != NULL' failed
5:14:05 AM: (sharp:1924): GLib-CRITICAL **: 13:14:05.269: g_hash_table_lookup: assertion 'hash_table != NULL' failed

Here's the full build logs.

samajammin commented 3 years ago

Still an issue.

samajammin commented 3 years ago

Solved in #2295

ethereum / ethereum-org-website

Fix Github API rate limits #2160