Possible to Throttle Image Downloads

[jonesbp]

I have an Airtable column for image attachments mapped as a fileNode in gatsby-config.js and am grabbing those images and performing a resize on them with sharp with no problems for small sets of images (e.g., 20 rows).

The build succeeds and the following GraphQL gives me the thumbnails I want:

products: allAirtable(filter: {table: {eq: "Prints"}}) {
  edges {
    node {
      data {
        Name
        Image_File {
          localFiles {
            childImageSharp {
              resize(width: 440) {
                src
              }
            }
          }
        }
        Reference
      }
    }
  }
}

However, when I throw my full set of production data (> 500 rows) the build will stall out (I have to ^C to kill it eventually) when I start getting errors from localFileCheck() in gatsby-source-airtable:

You specified a fileNode, but we caught an error. First check that you have gatsby-source-filesystem installed?
 TypeError: Cannot read property 'id' of undefined
at Promise.all.map.attachmentNode (/Users/jonesbp/src/store-demo/node_modules/gatsby-source-airtable/gatsby-node.js:281:42)

Each of which is preceded by a warning from gatsby-source-filesystem:

**Failed to process remote content https://dl.airtable.com/.attachments/6895bafdd284a8d711d89dc8440e549f/f6a8dc3c/2dI1uQ-JTcGJkNCp5ODKqAz-1-edit-1.jpeg**

Where the URL will be whatever URL is associated with that row of Airtable data. If I copy the URL to my browser, the image will load.

So my suspicion of what is happening is that I am (through gatsby-source-filesystem via gatsby-source-airtable) making too large a burst of HTTP requests and Airtable begins to block my requests. gatsby-source-filesystem then gives undefined data back to gatsby-source-airtable generating the error.

I have been trying to find the best place to add a throttle to gatsby-source-airtable to test this theory (and to mitigate this problem if the theory is correct), but so far have not been successful and feel like I need to ask for help from someone that understands how the remote requests work better than I do.

[jbolda]

We are using createRemoteFileNode which does all of the fetching for us at L269-L275. We then use Promise.all to let these process concurrently.

In reality though, if you have one picture per row, that means each node will have one picture, and we are running the Promise.all for each node (at L269-L275) because Airtable attachment fields house multiple attachments. (So throttling around those lines of code wouldn't do much, nor would adding throttling in createRemoteFileNode.)

I believe if you want to try running the downloading serially (or throttle in someway), you would need to do that around the node creation itself. See L158-167.

[jonesbp]

Thank you for the context, @jbolda. I will try starting there when I’m able to get back to this.

[jonesbp]

I've submitted a solution that uses the concurrency limit in Promise.map() from bluebird in #50