Feature/GitHub api call throttling

bcgov / gatsby-source-github-raw

A Gatsby Source Plugin that leverages Github V3 Contents API

Apache License 2.0

0 stars 1 forks source link

Feature/GitHub api call throttling #4

Closed sheaphillips closed 4 years ago

sheaphillips commented 4 years ago

Summary

We have been getting rate/abuse limited by Github lately due to the number of parallel api calls we are making. This PR introduces rate limiting / throttling using bottleneck to prevent this from happening.

Notable Changes

added bottleneck dependency
create bottleneck proxy/wrapper with a reasonable/recommended config to control number of calls
wrapping call to fetch

sheaphillips commented 4 years ago

@patricksimonian a couple of unit tests are failing locally. I took a quick look but fix was not immediately obvious due to lack of familiarity with code base. It probably will be obvious to you. Hoping you can fix it up. If not, LMK and I can sort it out.

sheaphillips commented 4 years ago

Note I have tested this locally with the DevHub (using npm link) and it works fine. Subjectively build time seems shorter...so maybe request stacking up was actually slowing things down...we'll see I guess.