Plugin runs multiple times

[jdahdah]

Hey, I'm running the plugin v2.3.5 with Gatsby v4.7.2 and I noticed something strange in my logs. I am using 4 instances of gatsby-source-custom-api as per the guidance in the README, all with a valid schema. And the strange thing is, unique file names (downloaded assets) show up 4 times as well. So it appears that if I have X instances of the plugin, it will run all of them for each instance. This is of course increasing build times and may be exacerbating this memory issue on Netlify for me. My config is as follows, is there anything I am doing wrong that would cause this? At first I thought it might be a general issue with my config, but it looks like this is the only plugin doing this.

Update: after more research on my local machine, it looks like it's more arbitrary than the number of instances. On a fast machine, file names are showing up 10x. I've turned off parallel sourcing and removed all page generation from gatsby-node.js to isolate this but it makes no difference.

Update 2: I've now run this same plugin v2.3.5 on the same project in a different branch still running Gatsby v3.14.5 and filenames only show up once. So there appears to be an incompatibility between this plugin and Gatsby 4.

    {
      resolve: "gatsby-source-custom-api",
      options: {
        rootKey: "jobCategoriesEu",
        url: "https://boards-api.greenhouse.io/v1/boards/xxxEu/departments.json",
        schemas: {
          jobCategoriesEu: `
              departments: [departments] @link(by: "id", from: "departments___NODE")
            `,
          departments: `
              department: [department] @link(by: "id", from: "department___NODE")
            `,
          department: `
              id: ID!
              name: String!
              parent_id: ID
              child_ids: [Int]
              jobs: [job] @link(by: "id", from: "job___NODE")
            `,
          job: `
              id: ID!
              title: String!
              absolute_url: String!
              internal_job_id: Int
              updated_at: String
              location: location @link(by: "id", from: "location___NODE")
              data_compliance: [compliance] @link(by: "id", from: "compliance___NODE")
            `,
          location: `
              name: String
            `,
          compliance: `
              type: String
              requires_consent: Boolean
              retention_period: String
            `,
        },
      },
    },
    {
      resolve: "gatsby-source-custom-api",
      options: {
        rootKey: "jobCategoriesUa",
        url: "https://boards-api.greenhouse.io/v1/boards/xxxUa/departments.json",
        schemas: {
          jobCategoriesUa: `
              departments: [departments] @link(by: "id", from: "departments___NODE")
            `,
          departments: `
              department: [department] @link(by: "id", from: "department___NODE")
            `,
          department: `
              id: ID!
              name: String!
              parent_id: ID
              child_ids: [Int]
              jobs: [job] @link(by: "id", from: "job___NODE")
            `,
          job: `
              id: ID!
              title: String!
              absolute_url: String!
              internal_job_id: Int
              updated_at: String
              location: location @link(by: "id", from: "location___NODE")
              data_compliance: [compliance] @link(by: "id", from: "compliance___NODE")
            `,
          location: `
              name: String
            `,
          compliance: `
              type: String
              requires_consent: Boolean
              retention_period: String
            `,
        },
      },
    },
    {
      resolve: "gatsby-source-custom-api",
      options: {
        rootKey: "pressReleases",
        url: "https://api.pr.co/v1/pressrooms/xxxxx/press_releases.json",
        schemas: {
          pressReleases: `
            data: [data] @link(by: "id", from: "data___NODE")
            paging: [paging] @link(by: "id", from: "paging___NODE")
          `,
          data: `
            press: [press] @link(by: "id", from: "press___NODE")
          `,
          press: `
            id: ID!
            pressroom_id: ID!
            title: String!
            subtitle: String
            release_date: String
            release_location: String
            language: String
            social_media_pitch: String
            summary: String
            body_html: String
            permalink: String
            type: String
            state: String
            short_link: String
            pdf: String
            show_in_timeline: Boolean
            content_as_json: String
            draft_content_as_json: String
            freeform_two: Boolean
            reading_time: Int
            updated_at: String
          `,
          paging: `
            page: Int
            limit: Int
            total: Int
          `,
        },
      },
    },
    {
      resolve: "gatsby-source-custom-api",
      options: {
        rootKey: "pressKits",
        url: "https://api.pr.co/v1/pressrooms/xxxx/presskits.json",
        imageKeys: ["original"],
        schemas: {
          pressKits: `
            data: [data] @link(by: "id", from: "data___NODE")
          `,
          data: `
            media_counts: [media_counts] @link(by: "id", from: "media_counts___NODE")
            paging: [paging] @link(by: "id", from: "paging___NODE")
            media: [media] @link(by: "id", from: "media___NODE")
          `,
          media_counts: `
            documents: Int
            images: Int
            sounds: Int
            videos: Int
            total: Int
          `,
          paging: `
            page: Int
            limit: Int
            total: Int
          `,
          media: `
            sizes: sizes @link(by: "id", from: "sizes___NODE")
            title: String
            file_size: Int
            content_type: String
          `,
          sizes: `
            original: original @link(by: "id", from: "original___NODE")
            medium: medium @link(by: "id", from: "medium___NODE")
            large: large @link(by: "id", from: "large___NODE")
            square: square @link(by: "id", from: "square___NODE")
          `,
          original: `
            url: String
            height: String
            width: String
          `,
          medium: `
            url: String
          `,
          large: `
            url: String
          `,
          square: `
            url: String
          `,
        },
      },
    },

[Tomas-Juri]

I tried to debug this issue locally - i cloned the repo and set up an example site according to https://www.gatsbyjs.com/docs/how-to/plugins-and-themes/creating-a-source-plugin/, linked the plugin to the example and added it to the plugin list as following:

{
  'resolve': require.resolve(`../`),
  options: {
    url: "__my_custom_api_url",
  },
}

And indeed, the plugin was loaded several times (a simple console.log at the start of the plugin revealed this).

Then i noticed that the plugin is exported as exports.createSchemaCustomization while the official documentation suggests exports.sourceNodes. Switching to the later fixed the issue for me.

I don't know if this is on purpose since I don't know the gatsby plugins that well, but i hope this might be helpfull.

[Tomas-Juri]

Okay, I think the change to exports.createSchemaCustomization might not be that intentional, it was introduced in https://github.com/AndreasFaust/gatsby-source-custom-api/pull/42.

I opened a pull request to fix this: https://github.com/AndreasFaust/gatsby-source-custom-api/pull/44.

[jdahdah]

@Tomas-Juri Amazing, thank you so much. Your PR seems to have done the trick: the plugin is only run once and the issue the previous PR was meant to fix remains fixed. Applied via patch-package for now until it's released.