octokit / graphql.js

GitHub GraphQL API client for browsers and Node
MIT License
464 stars 83 forks source link

Getting HttpError 403 when using in enterprise. #162

Closed w3nda closed 4 years ago

w3nda commented 4 years ago

Hey I am using github graphql for an interesting feature, I am fetching all repos in an organisation, then going over each repo and checking if a certain package is used.

This is enterprise github, also means that everything runs behind a vpc in aws. When I run the query locally, everything works fine, but once I run it in aws lambda, I get this error below, but the funny thing is that this error is only appearing after a few times that the query is running. so my query is using pagination, for some pages it will work, but then it will fail. Can you please explain what is wrong here? (limit is not the issue, I'm barely using 20 requests, limit is at 60000)

{
    "name": "HttpError",
    "status": 403,
    "headers": {
        "access-control-allow-origin": "*",
        "access-control-expose-headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type",
        "connection": "close",
        "content-encoding": "gzip",
        "content-security-policy": "default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.git.autodesk.com media.git.autodesk.com www.githubstatus.com git.autodesk.com; font-src assets.git.autodesk.com; form-action 'self' git.autodesk.com gist.git.autodesk.com; frame-ancestors 'none'; frame-src render.git.autodesk.com; img-src * data:; manifest-src 'self'; media-src 'none'; script-src assets.git.autodesk.com; style-src 'unsafe-inline' assets.git.autodesk.com",
        "content-type": "application/json; charset=utf-8",
        "date": "Sun, 02 Aug 2020 06:48:34 GMT",
        "gh-limited-by": "time-based",
        "gh-limited-group": "api",
        "referrer-policy": "origin-when-cross-origin, strict-origin-when-cross-origin",
        "retry-after": "60",
        "server": "GitHub.com",
        "status": "403 Forbidden",
        "strict-transport-security": "max-age=31536000; includeSubdomains",
        "transfer-encoding": "chunked",
        "x-content-type-options": "nosniff",
        "x-frame-options": "deny",
        "x-github-media-type": "github.v3; format=json",
        "x-github-request-id": "cf91f874-4f3e-452f-b3e1-288b01fff0c7",
        "x-runtime-rack": "0.015880",
        "x-xss-protection": "1; mode=block"
    },
    "request": {
        "method": "POST",
        "url": "https://git.autodesk.com/api/graphql",
        "headers": {
            "accept": "application/vnd.github.v3+json",
            "user-agent": "octokit-graphql.js/4.5.1 Node.js/10.21.0 (Linux 4.14; x64)",
            "authorization": "Bearer [REDACTED]",
            "content-type": "application/json; charset=utf-8"
        },
        "body": "{\"query\":\"\\nquery SearchMostTop10Star($queryString: String!, $pageSize: Int!) {\\n  rateLimit {\\n    limit\\n    cost\\n    remaining\\n    resetAt\\n  }\\n  search(query: $queryString, type: REPOSITORY, first: $pageSize , after: \\\"Y3Vyc29yOjcwMA==\\\") {\\n    pageInfo {\\n      endCursor\\n      hasNextPage\\n    }\\n    repositoryCount\\n    edges {\\n      node {\\n        ... on Repository {\\n          name\\n          url\\n          defaultBranchRef {\\n            name\\n            target {\\n              ... on Commit {\\n                tree {\\n                  entries {\\n                    name\\n                    object {\\n                      ... on Blob {\\n                        text\\n                      }\\n                      ... on Tree {\\n                        entries {\\n                          name\\n                          type\\n                          object {\\n                            ... on Blob {\\n                              text\\n                            }\\n                          }\\n                        }\\n                      }\\n                    }\\n                  }\\n                }\\n              }\\n            }\\n          }\\n        }\\n      }\\n    }\\n  }\\n}\\n\",\"variables\":{\"queryString\":\"org:BIM360\",\"pageSize\":100}}"
    },
    "documentation_url": "https://developer.github.com/v3/#abuse-rate-limits"
}

I also get this error

ERROR   { HttpError: You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.
    at response.text.then.message (/var/task/node_modules/@octokit/request/dist-node/index.js:66:23)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  name: 'HttpError',
  status: 403,
  headers:
   { 'access-control-allow-origin': '*',
     'access-control-expose-headers':
      'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type',
     connection: 'close',
     'content-encoding': 'gzip',
     'content-security-policy':
      'default-src \'none\'; base-uri \'self\'; block-all-mixed-content; connect-src \'self\' uploads.git.autodesk.com media.git.autodesk.com www.githubstatus.com git.autodesk.com; font-src assets.git.autodesk.com; form-action \'self\' git.autodesk.com gist.git.autodesk.com; frame-ancestors \'none\'; frame-src render.git.autodesk.com; img-src * data:; manifest-src \'self\'; media-src \'none\'; script-src assets.git.autodesk.com; style-src \'unsafe-inline\' assets.git.autodesk.com',
     'content-type': 'application/json; charset=utf-8',
     date: 'Sun, 02 Aug 2020 07:19:26 GMT',
     'gh-limited-by': 'time-based',
     'gh-limited-group': 'api',
     'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
     'retry-after': '60',
     server: 'GitHub.com',
     status: '403 Forbidden',
     'strict-transport-security': 'max-age=31536000; includeSubdomains',
     'transfer-encoding': 'chunked',
     'x-content-type-options': 'nosniff',
     'x-frame-options': 'deny',
     'x-github-media-type': 'github.v3; format=json',
     'x-github-request-id': 'ccd7a053-e6af-4e64-9ae2-c2ce85343a24',
     'x-runtime-rack': '0.007449',
     'x-xss-protection': '1; mode=block' },
  request:
   { method: 'POST',
     url: 'https://git.autodesk.com/api/graphql',
     headers:
      { accept: 'application/vnd.github.v3+json',
        'user-agent': 'octokit-graphql.js/4.5.1 Node.js/10.21.0 (Linux 4.14; x64)',
        authorization: 'Bearer [REDACTED]',
        'content-type': 'application/json; charset=utf-8' },
     body:
      '{"query":"\\nquery SearchMostTop10Star($queryString: String!, $pageSize: Int!) {\\n  rateLimit {\\n    limit\\n    cost\\n    remaining\\n    resetAt\\n  }\\n  search(query: $queryString, type: REPOSITORY, first: $pageSize , after: \\"Y3Vyc29yOjgwMA==\\") {\\n    pageInfo {\\n      endCursor\\n      hasNextPage\\n    }\\n    repositoryCount\\n    edges {\\n      node {\\n        ... on Repository {\\n          name\\n          url\\n          defaultBranchRef {\\n            name\\n            target {\\n              ... on Commit {\\n                tree {\\n                  entries {\\n                    name\\n                    object {\\n                      ... on Blob {\\n                        text\\n                      }\\n                      ... on Tree {\\n                        entries {\\n                          name\\n                          type\\n                          object {\\n                            ... on Blob {\\n                              text\\n                            }\\n                          }\\n                        }\\n                      }\\n                    }\\n                  }\\n                }\\n              }\\n            }\\n          }\\n        }\\n      }\\n    }\\n  }\\n}\\n","variables":{"queryString":"org:BIM360","pageSize":100}}' },
  documentation_url: 'https://developer.github.com/v3/#abuse-rate-limits' }
w3nda commented 4 years ago

Solved using retry mechanism using the Retry-After response header I get from github.