gatsby-source-drupal: Incremental build - adding a backlink to a node is not enough to rerun page queries related to it

[kszot-ref]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• [X] This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

Example scenario:

• You source your data from drupal JSON:API.
• You get two nodes: "node--vacancy" and "node--office".
• In general there could exist a relationship between those two ("node--vacancy" could be related to a "node--office"), but currently there's no connection between those in drupal.
• You run a full build ("npm run build").
• Now you update the "node--vacancy" in drupal, so that it is connected to the "node--office" node. This information will appear in the Gatsby Fastbuild Log.
• You run a new production build ("npm run build") with fastbuilds enabled.
• You get the new "node--vacancy" node from the fastbuild endpoint, it contains a relationship to the "node--office" node.
• During the update, a backlink is created, so that "node--office" also has a relationship to the "node--vacancy". (You can connect multiple vacancies to offices, so this is an array of IDs, in this case with just one record).
• Both "node--office" node and "node--vacancy" are passed on as arguments to "createNode" method, with an updated contentDigest.
• All queries that use vacancies ("allNodeVacancy" and "nodeVacancy" for that specific vacancy) are rerun.
• Office queries are not rerun ("allNodeOffice" and "nodeOffice" for that specific office).

The end result is that page-data.json for that page (office) has stale data, so it's not rebuilt properly.

In our use case the backlinks are used to display a list of vacancies on an office page, and although the backlink is created fine in the plugin, that's not enough to rebuild the page, so altough the graphql data is updated properly, the page uses old data.

I was able to check the node before it was passed on to the "createNode" function with a debugger. Everything seemed fine there. But even manually changing the value of one of the fields (before the contentDigest is genererated) didn't cause page query reruns. I suspect this might be an issue that instead of creating a new object with copied data, the plugin is modifying an object that it got through the "getNode" function. After this I've updated this "node--office" in drupal (didn't change any data, just saved it), and the next build picked that up properly and rerun the queries - the data didn't change, but the "createNode" method got a different object with the same fields which was enough to cause an update.

Reproduction Link

-

Expected Result

Office queries would be rerun.

Actual Result

Office queries aren't rerun.

Environment

System:
    OS: Linux 5.4 Ubuntu 20.04.2 LTS (Focal Fossa)
    CPU: (8) x64 Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 14.17.3 - ~/.nvm/versions/node/v14.17.3/bin/node
    Yarn: 1.22.10 - /mnt/c/Program Files/nodejs/yarn
    npm: 6.14.13 - ~/.nvm/versions/node/v14.17.3/bin/npm
  npmPackages:
    gatsby: ^3.10.2 => 3.10.2
    gatsby-background-image: ^1.5.3 => 1.5.3
    gatsby-plugin-create-client-paths: ^3.10.0 => 3.10.0
    gatsby-plugin-emotion: ^6.10.0 => 6.10.0
    gatsby-plugin-google-tagmanager: ^3.10.0 => 3.10.0
    gatsby-plugin-image: ^1.10.1 => 1.10.1
    gatsby-plugin-loadable-components-ssr: ^3.2.0 => 3.2.0
    gatsby-plugin-manifest: ^3.10.0 => 3.10.0
    gatsby-plugin-meta-redirect: ^1.1.1 => 1.1.1
    gatsby-plugin-react-helmet: ^4.10.0 => 4.10.0
    gatsby-plugin-react-helmet-canonical-urls: ^1.4.0 => 1.4.0
    gatsby-plugin-robots-txt: ^1.6.8 => 1.6.8
    gatsby-plugin-sass: ^4.10.0 => 4.10.0
    gatsby-plugin-sharp: ^3.10.2 => 3.10.2
    gatsby-plugin-sitemap: ^4.6.0 => 4.6.0
    gatsby-plugin-typegen: ^2.2.4 => 2.2.4
    gatsby-plugin-use-query-params: ^1.0.1 => 1.0.1
    gatsby-plugin-webpack-bundle-analyser-v2: ^1.1.24 => 1.1.24
    gatsby-query-params: ^1.2.1 => 1.2.1
    gatsby-source-drupal: ^4.14.0 => 4.14.0
    gatsby-source-filesystem: ^3.10.0 => 3.10.0
    gatsby-transformer-sharp: ^3.10.0 => 3.10.0

Config Flags

No response

[kszot-ref]

We've went through the code with Kyle, so I've got a few tips where to look to debug this issue. During the call we didn't pinpoint it, but I've tried again the next day.

I've went through the node creation process with a debugger and came to a conclusion that this is indeed an issue inside the gatsby-source-drupal plugin.

To create a backlink for a referenced node, the plugin does the following steps:

• use "getNode" method to get the referenced node
• it adds the new backlink inplace to that referenced node
• it regenerates the content digest inplace
• it passes the node to "createNode" function
• inside of it, the digests are compared - this is where the issue occurs

https://github.com/gatsbyjs/gatsby/blob/08c5b2a55559fd1419222ae6cb5a7d3c1e2a098d/packages/gatsby/src/utils/nodes.ts#L9-L16

When the code reaches this point, it gets the node from the datastore, but the issue is that data was modified inplace, the updated values are already in there. So it actually compares the new contentDigest with the already updated contentDigest, which means that the code thinks that the node didn't update at all, as in this scenario they will always be the same.