GraphQL Error 'Invariant violation: inconsistent node counters detected' with gatsby-source-contentful downloadLocal option

afladmark commented 2 years ago

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
[X] This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

Now that the issues (https://github.com/gatsbyjs/gatsby/issues/33871) with gatsby-source-contentful's downloadLocal option have been resolved, we've upgraded to Gatsby 4. However, we're running into a different but completely debilitating issue which is breaking the vast majority of our Gatsby Cloud Preview builds, incremental production builds and which surfaces while navigating in Gatsby develop as well.

On a clean build, everything seems to work fine in all environments. On develop, stopping and restarting the develop process one or two times and then clicking around triggers the error. On Gatsby Cloud, "clear cache and (re)build succeeds", but incremental builds fail almost 100% of the time. Gatsby Preview is entirely useless. We're stuck clearing cache and rebuilding constantly. For us this takes 20-40 mins.

 ERROR #85901  GRAPHQL

There was an error in your GraphQL query:

Invariant violation: inconsistent node counters detected

  45 |     }
  46 |   }
  47 |   childContentfulBlogPostContentTextNode {
  48 |     childMarkdownRemark {
  49 |       timeToRead
  50 |     }
  51 |   }
  52 | }
  53 |
  54 | fragment BlogCta on Query {
> 55 |   blogCtaCardImage: contentfulAsset(
     |   ^
  56 |     contentful_id: {eq: "3RwyPsi3e24HPSwglySxnI"}
  57 |     node_locale: {eq: $locale}
  58 |   ) {
  59 |     title
  60 |     localFile {
  61 |       childImageSharp {
  62 |         gatsbyImageData(width: 280)
  63 |       }
  64 |     }
  65 |   }

File path: /Users/andrew/Repositories/www/src/templates/Blog.js
Url path: /blog/
Plugin: none

  Error: Invariant violation: inconsistent node counters detected

  - indexing.ts:986 intersectNodesByCounter
    [www]/[gatsby]/src/datastore/in-memory/indexing.ts:986:15

  - run-fast-filters.ts:106 applyFastFilters
    [www]/[gatsby]/src/datastore/in-memory/run-fast-filters.ts:106:30

  - run-fast-filters.ts:347 convertAndApplyFastFilters
    [www]/[gatsby]/src/datastore/in-memory/run-fast-filters.ts:347:18

  - run-fast-filters.ts:282 runFastFiltersAndSort
    [www]/[gatsby]/src/datastore/in-memory/run-fast-filters.ts:282:18

  - lmdb-datastore.ts:176 Object.runQuery
    [www]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:176:26

  - node-model.js:347 LocalNodeModel._query
    [www]/[gatsby]/src/schema/node-model.js:347:58

  - runMicrotasks

  - task_queues.js:95 processTicksAndRejections
    internal/process/task_queues.js:95:5

  - node-model.js:415 LocalNodeModel.findOne
    [www]/[gatsby]/src/schema/node-model.js:415:34

  - async Promise.all

  - graphql-runner.ts:220 GraphQLRunner.query
    [www]/[gatsby]/src/query/graphql-runner.ts:220:14

  - query-runner.ts:135 queryRunner
    [www]/[gatsby]/src/query/query-runner.ts:135:14

I've had to rollback to Gatsby 3 which is soul destroying after this much time.

I'm happy to do literally anything I can to help get this resolved. I can point you to any of our cloud builds or share source privately. If you have advice on how to create a minimal reproduction with Contentful, I can try to do that as well. Are there some publicly accessible Contentful instances/guest credentials that can be used?

For my part, I've stripped everything out of our config and site I possibly can and the error persists.

Reproduction Link

See comment below: https://github.com/gatsbyjs/gatsby/issues/34616#issuecomment-1029461729

Steps to Reproduce

Setup gatsby-source-contentful with the downloadLocal option enabled.
Reference a ContentfulAsset in graphql (all of ours use the localFile node as above in error)
Run gatsby develop, click around a bit.
Stop and restart gatsby develop (once or twice seems to guarantee it).
Refresh or re-navigate to a page referencing the ContentfulAsset.

Alternatively, a Gatsby Preview update triggered by a Preview Webhook fails 100% of the time (except when it randomly decides to run a full build after enough failures).

Expected Result

It shouldn't error and all images should be presented.

Actual Result

GraphQL Error: Invariant violation: inconsistent node counters detected error is displayed both in console and in gatsby develop pop over. For Preview and incremental builds the build fails with this same error.

Environment

System:
    OS: macOS 12.3
    CPU: (10) x64 Apple M1 Max
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.17.5 - ~/.nvm/versions/node/v14.17.5/bin/node
    Yarn: 1.22.17 - /opt/homebrew/bin/yarn
    npm: 6.14.14 - ~/.nvm/versions/node/v14.17.5/bin/npm
  Browsers:
    Chrome: 97.0.4692.99
    Firefox: 96.0.3
    Safari: 15.4
  npmPackages:
    gatsby: ^4.5.0 => 4.6.0 
    gatsby-plugin-catch-links: ^4.5.0 => 4.6.0 
    gatsby-plugin-env-variables: ^2.2.0 => 2.2.0 
    gatsby-plugin-graphql-codegen: ^3.1.0 => 3.1.0 
    gatsby-plugin-image: ^2.5.1 => 2.6.0 
    gatsby-plugin-loadable-components-ssr: 4.1.1 => 4.1.1 
    gatsby-plugin-lodash: ^5.5.0 => 5.6.0 
    gatsby-plugin-mailchimp: ^5.2.2 => 5.2.2 
    gatsby-plugin-react-helmet: ^5.4.0 => 5.6.0 
    gatsby-plugin-react-svg: ^3.1.0 => 3.1.0 
    gatsby-plugin-robots-txt: ^1.6.14 => 1.7.0 
    gatsby-plugin-sharp: ^4.5.1 => 4.6.0 
    gatsby-plugin-styled-components: ^5.5.0 => 5.6.0 
    gatsby-remark-autolink-headers: ^5.5.0 => 5.6.0 
    gatsby-source-contentful: ^7.3.0 => 7.4.0 
    gatsby-source-filesystem: ^4.5.1 => 4.6.0 
    gatsby-transformer-remark: ^5.5.1 => 5.6.0 
    gatsby-transformer-sharp: ^4.5.0 => 4.6.0

Config Flags

N/A

wardpeet commented 2 years ago

Hi,

is it possible to remove node_modules and your lock file and re-install all the things? If you're running yarn you could do the following. npx yarn-deduplicate yarn.lock && yarn This should make sure, nested deps are deduped.

afladmark commented 2 years ago

Hi @wardpeet! Thank you so much for your reply. I've just spent a couple hours on this. I tested with both yarn and npm. I also ran handy tools like yarn dlx @yarnpkg/doctor to see if I could see any additional issues. There was a lot of crap relating to the now abandoned gatsby-plugin-graphql-codegen, so I've just removed that from the project entirely. Beyond that, there doesn't seem to be any other package issues:

andrew@MacBook-Pro www % yarn dlx @yarnpkg/doctor
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 4s 999ms
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0000: └ Completed
➤ YN0000: Done with warnings in 5s 250ms

➤ YN0000: Found 1 package(s) to process
➤ YN0000: For a grand total of 186 file(s) to validate

➤ YN0000: ┌ /Users/andrew/Repositories/www/package.json
➤ YN0000: │ /Users/andrew/Repositories/www/jest.config.js:9:28: Strings should avoid referencing the node_modules directory (prefer require.resolve)
➤ YN0000: │ /Users/andrew/Repositories/www/jest.config.js:10:29: Strings should avoid referencing the node_modules directory (prefer require.resolve)
➤ YN0000: │ /Users/andrew/Repositories/www/jest.config.js:11:29: Strings should avoid referencing the node_modules directory (prefer require.resolve)
➤ YN0000: │ /Users/andrew/Repositories/www/package.json:17:28: Unmet transitive peer dependency on core-js@^3.0.0, via babel-preset-gatsby@^2.5.1
➤ YN0000: │ /Users/andrew/Repositories/www/package.json:35:33: Unmet transitive peer dependency on sharp@^0.29.3, via gatsby-source-contentful@^7.3.0
➤ YN0000: │ /Users/andrew/Repositories/www/package.json:50:26: Unmet transitive peer dependency on react-is@>= 16.8.0, via styled-components@^5.3.3
➤ YN0000: │ /Users/andrew/Repositories/www/package.json:93:39: Unmet transitive peer dependency on webpack@>=1.11.0, via eslint-import-resolver-webpack@^0.13.2
➤ YN0000: └ Completed in 36s 289ms
➤ YN0000: ⠏ --------------------------------------------------------------------------------

➤ YN0000: Failed with errors in 36s 291ms

In either scenario, with yarn or npm and with new lockfiles in both cases I get the same result.

I definitely understand where you're coming from. We ran into several odd graphql errors on upgrading to Gatsby 4 which were resolved by blowing away package-lock.json and node_modules as you suggested. But this one persists...

The question in my mind, if this were a package issue why would it only occur on subsequent runs for develop and only on incremental builds (locally and on gatsby cloud)? And only on ContentfulAsset nodes?

Is there anything I can try to do to help narrow it down? When I had a read of the code related to the stacktrace it seemed to be related to checking data in the internal nodes, but was all a bit opaque to me.

axe312ger commented 2 years ago

@afladmark I'll try to reproduce it today. If I got no luck with that, I might need access to your project. I'll keep u updated!

axe312ger commented 2 years ago

@afladmark it does not happen anymore. I reproduced the same situation I had in December.

This makes me wonder as well, if this is a dependency duplication issue.

Can you please post the output of yarn why gatsby-plugin-utils and yarn why gatsby-core-utils?

afladmark commented 2 years ago

Thank you for looking into it @axe312ger. Here is the output of both those commands:

andrew@MacBook-Pro www % yarn why gatsby-plugin-utils
yarn why v1.22.15
[1/4] 🤔  Why do we have the module "gatsby-plugin-utils"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found "gatsby-plugin-utils@3.0.0"
info Reasons this module exists
   - "gatsby" depends on it
   - Hoisted from "gatsby#gatsby-plugin-utils"
   - Hoisted from "gatsby-plugin-sharp#gatsby-plugin-utils"
   - Hoisted from "gatsby-source-contentful#gatsby-plugin-utils"
   - Hoisted from "gatsby#gatsby-plugin-page-creator#gatsby-plugin-utils"
info Disk size without dependencies: "304KB"
info Disk size with unique dependencies: "1.71MB"
info Disk size with transitive dependencies: "2.22MB"
info Number of shared dependencies: 5
✨  Done in 0.31s.
andrew@MacBook-Pro www % yarn why gatsby-core-utils
yarn why v1.22.15
[1/4] 🤔  Why do we have the module "gatsby-core-utils"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found "gatsby-core-utils@3.6.0"
info Reasons this module exists
   - "babel-preset-gatsby" depends on it
   - Hoisted from "babel-preset-gatsby#gatsby-core-utils"
   - Hoisted from "gatsby#gatsby-core-utils"
   - Hoisted from "gatsby-plugin-image#gatsby-core-utils"
   - Hoisted from "gatsby-plugin-sharp#gatsby-core-utils"
   - Hoisted from "gatsby-source-contentful#gatsby-core-utils"
   - Hoisted from "gatsby-source-filesystem#gatsby-core-utils"
   - Hoisted from "gatsby-transformer-remark#gatsby-core-utils"
   - Hoisted from "gatsby#babel-plugin-remove-graphql-queries#gatsby-core-utils"
   - Hoisted from "gatsby#gatsby-cli#gatsby-core-utils"
   - Hoisted from "gatsby#gatsby-plugin-page-creator#gatsby-core-utils"
   - Hoisted from "gatsby#gatsby-telemetry#gatsby-core-utils"
   - Hoisted from "gatsby#gatsby-plugin-page-creator#gatsby-page-utils#gatsby-core-utils"
info Disk size without dependencies: "1.07MB"
info Disk size with unique dependencies: "2.89MB"
info Disk size with transitive dependencies: "5.05MB"
info Number of shared dependencies: 50
✨  Done in 0.34s.

afladmark commented 2 years ago

And with Yarn 3 as well:

andrew@MacBook-Pro www % yarn why gatsby-plugin-utils               
├─ gatsby-plugin-page-creator@npm:4.6.0
│  └─ gatsby-plugin-utils@npm:3.0.0 (via npm:^3.0.0)
│
├─ gatsby-plugin-page-creator@npm:4.6.0 [1e3e2]
│  └─ gatsby-plugin-utils@npm:3.0.0 [19a50] (via npm:^3.0.0 [19a50])
│
├─ gatsby-plugin-sharp@npm:4.6.0
│  └─ gatsby-plugin-utils@npm:3.0.0 (via npm:^3.0.0)
│
├─ gatsby-plugin-sharp@npm:4.6.0 [b3b7e]
│  └─ gatsby-plugin-utils@npm:3.0.0 [19a50] (via npm:^3.0.0 [19a50])
│
├─ gatsby-source-contentful@npm:7.4.0
│  └─ gatsby-plugin-utils@npm:3.0.0 (via npm:^3.0.0)
│
├─ gatsby-source-contentful@npm:7.4.0 [b3b7e]
│  └─ gatsby-plugin-utils@npm:3.0.0 [19a50] (via npm:^3.0.0 [19a50])
│
├─ gatsby@npm:4.6.0
│  └─ gatsby-plugin-utils@npm:3.0.0 (via npm:^3.0.0)
│
└─ gatsby@npm:4.6.0 [b3b7e]
   └─ gatsby-plugin-utils@npm:3.0.0 [19a50] (via npm:^3.0.0 [19a50])
andrew@MacBook-Pro www % yarn why gatsby-core-utils                 
├─ babel-plugin-remove-graphql-queries@npm:4.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ babel-plugin-remove-graphql-queries@npm:4.6.0 [1e3e2]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ babel-plugin-remove-graphql-queries@npm:4.6.0 [b5e53]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ babel-preset-gatsby@npm:2.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ babel-preset-gatsby@npm:2.6.0 [1e3e2]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ babel-preset-gatsby@npm:2.6.0 [b3b7e]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-cli@npm:4.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-page-utils@npm:2.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-plugin-image@npm:2.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-plugin-image@npm:2.6.0 [b3b7e]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-plugin-page-creator@npm:4.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-plugin-page-creator@npm:4.6.0 [1e3e2]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-plugin-sharp@npm:4.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-plugin-sharp@npm:4.6.0 [b3b7e]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-source-contentful@npm:7.4.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-source-contentful@npm:7.4.0 [b3b7e]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-source-filesystem@npm:4.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-source-filesystem@npm:4.6.0 [b3b7e]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-telemetry@npm:3.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-transformer-remark@npm:5.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby-transformer-remark@npm:5.6.0 [b3b7e]
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
├─ gatsby@npm:4.6.0
│  └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
│
└─ gatsby@npm:4.6.0 [b3b7e]
   └─ gatsby-core-utils@npm:3.6.0 (via npm:^3.6.0)
andrew@MacBook-Pro www %

axe312ger commented 2 years ago

@wardpeet this does not look like duplicated to me? What do you think?

afladmark commented 2 years ago

Hi @wardpeet and @axe312ger,

I've managed to create a minimum reproduction of this issue and some steps to reproduce.

You can fetch the repo (based on gatsby-starter-minimal) here: https://github.com/afladmark/gatsby-34616-repro/

There are two pages linked to each other. A cat and a dog. Each with a ContentfulAsset image on it.

Steps to repro:

npm run develop
Click back and forth between the cat and dog. All is well.
Kill gatsby develop (I usually just ctrl+c).
Make a change to BOTH pages such that Gatsby will recognize the pages have changed and need rebuilding. Editing the graphql query by adding the title field seems to trigger the issue 100% of the time, like so:

export const query = graphql`
  query {
    funnyDog: contentfulAsset(
      contentful_id: { eq: "1EVcdvsE44KC8okqnQnuaP" }
      node_locale: { eq: "en-US" }
    ) {
      title
      localFile {
        childImageSharp {
          gatsbyImageData(width: 600)
        }
      }
    }
  }
`;

Restart gatsby develop with npm run develop
Load the site, or refresh existing open site, or click to navigate to the other page
Boom:

 ERROR #85901  GRAPHQL

There was an error in your GraphQL query:

Invariant violation: inconsistent node counters detected

   1 | query pageUsersandrewRepositoriesgatsby34616ReprosrcpagesindexJs3411352589 {
>  2 |   funnyCat: contentfulAsset(
     |   ^
   3 |     contentful_id: {eq: "3CVwy2kmNLfP7G8tplGGyS"}
   4 |     node_locale: {eq: "en-US"}
   5 |   ) {
   6 |     title
   7 |     localFile {
   8 |       childImageSharp {
   9 |         gatsbyImageData(width: 600)
  10 |       }
  11 |     }
  12 |   }

File path: /Users/andrew/Repositories/gatsby-34616-repro/src/pages/index.js
Url path: /
Plugin: none

  Error: Invariant violation: inconsistent node counters detected

  - indexing.ts:986 intersectNodesByCounter
    [gatsby-34616-repro]/[gatsby]/src/datastore/in-memory/indexing.ts:986:15

  - run-fast-filters.ts:106 applyFastFilters
    [gatsby-34616-repro]/[gatsby]/src/datastore/in-memory/run-fast-filters.ts:106:30

  - run-fast-filters.ts:347 convertAndApplyFastFilters
    [gatsby-34616-repro]/[gatsby]/src/datastore/in-memory/run-fast-filters.ts:347:18

  - run-fast-filters.ts:282 runFastFiltersAndSort
    [gatsby-34616-repro]/[gatsby]/src/datastore/in-memory/run-fast-filters.ts:282:18

  - lmdb-datastore.ts:176 Object.runQuery
    [gatsby-34616-repro]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:176:26

  - node-model.js:347 LocalNodeModel._query
    [gatsby-34616-repro]/[gatsby]/src/schema/node-model.js:347:58

  - task_queues:96 processTicksAndRejections
    node:internal/process/task_queues:96:5

  - node-model.js:415 LocalNodeModel.findOne
    [gatsby-34616-repro]/[gatsby]/src/schema/node-model.js:415:34

  - async Promise.all

  - graphql-runner.ts:220 GraphQLRunner.query
    [gatsby-34616-repro]/[gatsby]/src/query/graphql-runner.ts:220:14

  - query-runner.ts:135 queryRunner
    [gatsby-34616-repro]/[gatsby]/src/query/query-runner.ts:135:14

If Gatsby doesn't think the page needs rebuilding the issue won't occur, but when it rebuilds the pages we get this error 100% of the time. Running gatsby clean will make it work again until the change occurs again... then it will explode.

Hopefully this works for you as well and helps to lead you to a fix. Please let me know if I can help in any other way.

axe312ger commented 2 years ago

Awesome! Will have a look next week!

axe312ger commented 2 years ago

@afladmark thank you for the steps to reproduce and the repo.

I can confirm that it is happening on my machine as well.

I'll contact the team and see what we can do!

wardpeet commented 2 years ago

I've debugged this a little bit more and contentful gives us the same asset twice which is a bit odd, same ID etc. Do you have two files that are exactly identical in your space? @axe312ger unsure if we should guard against this in the contentful plugin or if this is something that hardly happens.

{
  "assetItem": {
    "metadata": {
      "tags": []
    },
    "sys": {
      "space": {
        "sys": {
          "type": "Link",
          "linkType": "Space",
          "id": "x5hm9op8o2rf"
        }
      },
      "id": "1EVcdvsE44KC8okqnQnuaP",
      "type": "Asset",
      "createdAt": "2022-02-03T18:27:25.383Z",
      "updatedAt": "2022-02-03T18:27:25.383Z",
      "environment": {
        "sys": {
          "id": "master",
          "type": "Link",
          "linkType": "Environment"
        }
      },
      "revision": 1
    },
    "fields": {
      "title": {
        "en-US": "Funny Dog"
      },
      "file": {
        "en-US": {
          "url":
"//images.ctfassets.net/x5hm9op8o2rf/1EVcdvsE44KC8okqnQnuaP/d8da5002a906053ceac68aea71ab6177/pexels-pixabay-160846.jpg",  
          "details": {
            "size": 2297539,
            "image": {
              "width": 4728,
              "height": 3264
            }
          },
          "fileName": "pexels-pixabay-160846.jpg",
          "contentType": "image/jpeg"
        }
      }
    }
  }
}
// 
{
  "assetItem": {
    "metadata": {
      "tags": []
    },
    "sys": {
      "space": {
        "sys": {
          "type": "Link",
          "linkType": "Space",
          "id": "x5hm9op8o2rf"
        }
      },
      "id": "1EVcdvsE44KC8okqnQnuaP",
      "type": "Asset",
      "createdAt": "2022-02-03T18:27:25.383Z",
      "updatedAt": "2022-02-03T18:27:25.383Z",
      "environment": {
        "sys": {
          "id": "master",
          "type": "Link",
          "linkType": "Environment"
        }
      },
      "revision": 1
    },
    "fields": {
      "title": {
        "en-US": "Funny Dog"
      },
      "file": {
        "en-US": {
          "url":
"//images.ctfassets.net/x5hm9op8o2rf/1EVcdvsE44KC8okqnQnuaP/d8da5002a906053ceac68aea71ab6177/pexels-pixabay-160846.jpg",  
          "details": {
            "size": 2297539,
            "image": {
              "width": 4728,
              "height": 3264
            }
          },
          "fileName": "pexels-pixabay-160846.jpg",
          "contentType": "image/jpeg"
        }
      }
    }
  }
}

afladmark commented 2 years ago

Thanks for looking at it @wardpeet. In terms of the Contentful Space, this is a new, blank test Space we created which literally contains only two Assets, the cat and the dog images. Nothing else. No duplications on that side.

One thing that may be specific to our environment/config is that we use multiple locales. In this test Space, I've enabled English (US) and Spanish. However, we DO NOT use fallbacks. So if an asset or field doesn't contain a translation we get nulls, not the default locale's values. We do this so we can selectively fallback ourselves via custom resolvers or ignore/hide untranslated content.

We normally see duplicated contentful_ids for all translated variants of an object, but each normally has a unique id value. We've never observed duplication of ids in practice before.

axe312ger commented 2 years ago

This looks to me like an issue to the i18n implementation of the plugin.

I'll check that out!

axe312ger commented 2 years ago

@afladmark I wondered why I could not replicate when using the latest Gatsby code from master. Looks like the refactoring in 7.5.0 fixed the issue.

Try gatsby-source-contentful@7.5.0.

I found some small quirks in the implementation but the actual issue should be already fixed in the latest version :)

afladmark commented 2 years ago

@axe312ger Hmmm. That would be good news. I did install it last week and noticed I couldn't readily reproduce it. However, I believe when I asked others on the team to try it we did eventually run into the issue again but I don't know exactly how or under what circumstances.

I'll try it again today and see if I can confirm it one way or the other for you.

axe312ger commented 2 years ago

@afladmark perfect!

afladmark commented 2 years ago

Hi @axe312ger. Good news to report. We've been unable to reproduce this issue in our development environments and we've put latest back on production for a few days and our incremental Gatsby Preview builds are working there as well. Down from ~10 mins on 3.x to <2 mins on 4. So it looks like it has been resolved and we can FINALLY upgrade to 4!

Thank you so much for all your work on the Contentful source plugin. It's clearly going in the right direction and we're excited for the next major version changes as well. Happy to alpha/beta test anything.

One new issue we noticed is that if we make a non-breaking schema addition in Contentful (for example adding a field to an existing content type) the incremental Gatsby Preview builds fail with an error like "Encountered an error trying to infer a GraphQL type" until we force a restart/full build. I'm pretty sure that'll be resolved with the work you're doing to pull the schema, but thought I'd mention in case you wanted me to create another issue for that.

chunksnbits commented 2 years ago

Hi, and thanks for your efforts. Unfortunately we are still seeing the error, event after updating to 7.5.0 (or even 7.6.0, 7.7.0-alpha):

In our case we have enabled two locales in Contentful ("en-US", "de-DE") and we do see a lot of duplicate contentful_ids, and use some of them, as we localize out content within the entity to allow references.

We filter all our queries by locale though. So basically we should never run into a case where multiple entities with the same id are returned in a collection.

Is this still being worked on?

Leksat commented 2 years ago

@chunksnbits could you check if removing the sort solve the problem? Why I'm asking:

We recently updated Gatsby from 4.7.2 to 4.8.0 and got the same error as you. But in our case Gatsby is connected to Drupal (instead of Contentful) via a custom integration.

Invariant violation: inconsistent node counters detected

  213 | query Home($locale: String!, $remoteId: String!) {
  214 |   site {
  215 |     siteMetadata {
  216 |       title
  217 |     }
  218 |   }
> 219 |   contentHubItems: allDrupalPage(
      |   ^
  220 |     filter: {langcode: {eq: $locale}, contentHubMetadata: {type: {id: {glob: "*"}}}}
  221 |     sort: {fields: contentHubMetadata___date, order: DESC}
  222 |     limit: 4
  223 |   ) {
  224 |     nodes {
  225 |       ...ContentHubWidgetItem

I suspected that something can be wrong with contentHubMetadata___date sorting. Because previously this field was marked as deprecated (but now it is not 🤷). So I tried to remove it and the error disappeared.

And in the changelog record for Gatsby 4.8.0 I found

Make filter/sort query only hold onto node properties it needs #34747

which sound like something that can be related.

Also

I see the error only with gatsby build
With gatsby develop the page using the query works just fine
I can run the query without any issues on /___graphql

MikeBroders commented 2 years ago

hey, i ran into the same issue on gatsby 4.8.1

i'm using gatsby-source-airtable and had a sort filter in the erroring query

by removing the sort filter the error disappeared. and the build was successful.

so it seams that this issue is not directly related to the cms you are sourcing from.

will check if something changes if i go on gatsby 4.9.0

LekoArts commented 2 years ago

Please follow https://github.com/gatsbyjs/gatsby/issues/34910 for a fix on this.

gatsbyjs / gatsby