Gatsby transform sharp throws intermittent errors

seungpark commented 8 months ago

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
[X] This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

Gatsby transformer sharp throws intermittent errors during the build schema process:

 ERROR #11321  API.NODE.EXECUTION

"gatsby-transformer-sharp" threw an error while running the onCreateNode
lifecycle:

Cannot set properties of undefined (setting 'dirty')

TypeError Cannot set properties of undefined (setting 'dirty')
incrementalReducer
[gatsby]/src/redux/reducers/inference-metadata.ts:95:24
inference-metadata.ts
inferenceMetadataReducer [gatsby]/src/redux/reducers/inference-metadata.ts:140:23
 redux.js:568 combination
  [snooty]/[redux]/lib/redux.js:568:29
 redux.js:296 next
  [snooty]/[redux]/lib/redux.js:296:22
 index.ts:72 
  [snooty]/[gatsby]/src/redux/index.ts:72:68
 index.js:27 
  [snooty]/[redux-thunk]/lib/index.js:27:16
 redux.js:586 boundActionCreator
  [snooty]/[redux]/lib/redux.js:586:12
 api-runner-node.js:104 doubleBoundActionCreators.<computed>
  [snooty]/[gatsby]/src/utils/api-runner-node.js:104:20
 on-node-create.js:40 Object.onCreateNode
  [snooty]/[gatsby-transformer-sharp]/on-node-create.js:40:3
 api-runner-node.js:487 runAPI
  [snooty]/[gatsby]/src/utils/api-runner-node.js:487:37

[2K[1A[2K[Gnot finished building schema - 0.453s

Accompanying gatsby-config plugins:

const plugins = [
  `gatsby-plugin-image`,
  `gatsby-plugin-sharp`,
  {
    resolve: 'gatsby-source-filesystem',
    options: {
      name: 'images',
      path: `${__dirname}/src/images`,
    },
  },
  `gatsby-transformer-sharp`, // Needed for dynamic images
  'gatsby-plugin-emotion',
];

In gatsby-node.js:

const saveFiles = async (assets) => (
   Promise.all(<save asset promise to src/images>)
);

exports.sourceNodes = async ({ actions, createContentDigest, createNodeId }) => {
   ...
   await saveFiles(assets);
   ...
})

This error is not consistent, and it seems to be complaining about a ADD_CHILD_TO_NODE action.type case, at this line.

However, when debugging the build locally, the build always succeeds and I am able to see the ImageSharp nodes (as children of File nodes) in the graphql explorer.

Reproduction Link

https://github.com/seungpark/gatsby-reproduction

Steps to Reproduce

Define the gatsby config as outlined in above
Have an asynchronous process that saves image files to the directory that is sourced from the gatsby-source-filesystem plugin
Run gatsby build and look for error while in step source and transform nodes ...

Expected Result

I would expect this to be an error or success 100% of the time. Intermittent errors have been a pain for debugging. Ideally, this would be a successful Image Sharp node creation for each Image type file.

Actual Result

This error is thrown sometimes on the same build, but not 100%

Environment

  System:
    OS: Linux 5.10 Ubuntu 20.04.6 LTS (Focal Fossa)
    CPU: (4) x64 Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 18.19.0 - /usr/bin/node
    npm: 8.19.4 - /usr/bin/npm
  npmPackages:
    gatsby: ^5.0.0 => 5.7.0
    gatsby-plugin-emotion: ^8.0.0 => 8.7.0
    gatsby-plugin-google-tagmanager: ^5.0.0 => 5.7.0
    gatsby-plugin-image: ^3.13.0 => 3.13.0
    gatsby-plugin-layout: ^4.7.0 => 4.7.0
    gatsby-plugin-sharp: ^5.13.0 => 5.13.0
    gatsby-plugin-sitemap: ^6.0.0 => 6.7.0
    gatsby-source-filesystem: ^5.13.0 => 5.13.0
    gatsby-transformer-sharp: github:seungpark/gatsby-transformer-sharp =>5.13.0

Config Flags

No response

olehpratsko commented 8 months ago

Hey @seungpark , Your assets download script is not catching up with Gatsby nodes API at all. Use createRemoteFileNode instead. https://www.gatsbyjs.com/plugins/gatsby-source-filesystem/#createremotefilenode

exports.sourceNodes = async ({ actions, createNodeId, createContentDigest }) => {
  const { createNode } = actions;
  ...

    for (const image of remoteImages) {
      try {
        const fileNode = await createRemoteFileNode({
          url: image.url, // string that points to the URL of the image
          parentNodeId: node.id, // id of the parent node of the fileNode you are going to create
          createNode, // helper function in gatsby-node to generate the node
          createNodeId, // helper function in gatsby-node to generate the node id
          getCache,
        });
      } catch (error) {
        console.log(error);
      }
    }

remoteImages is just array of image urls

seungpark commented 7 months ago

Hey @seungpark , Your assets download script is not catching up with Gatsby nodes API at all. Use createRemoteFileNode instead. https://www.gatsbyjs.com/plugins/gatsby-source-filesystem/#createremotefilenode

exports.sourceNodes = async ({ actions, createNodeId, createContentDigest }) => {
  const { createNode } = actions;
  ...

    for (const image of remoteImages) {
      try {
        const fileNode = await createRemoteFileNode({
          url: image.url, // string that points to the URL of the image
          parentNodeId: node.id, // id of the parent node of the fileNode you are going to create
          createNode, // helper function in gatsby-node to generate the node
          createNodeId, // helper function in gatsby-node to generate the node id
          getCache,
        });
      } catch (error) {
        console.log(error);
      }
    }

remoteImages is just array of image urls

@olehpratsko instead of creating my own gatsby nodes, i am using gatsby filesystem to create image gatsby nodes of files where i save the assets

olehpratsko commented 7 months ago

@seungpark good workaround, I'll take it to the note. Thank you for sharing.

gatsbyjs / gatsby