Frequent build errors on Netlify

[marcoSven]

I experience intermittently build errors. They do not appear to be related to CMS content changes or code changes, it just randomly, unfortunately frequently, fails.

The following changes did not improve or worsen the fail rate - I think I removed netlify-plugin-gatsby-cache - #44 I removed gatsby-rehype-inline-images - #202

Clearing the cache before a deploy is not helping in my opinion.

The error is always something like this ... 11:17:15 AM: success run static queries - 0.204s - 16/16 78.48/s 11:17:18 AM: error [object Object] ENOENT: no such file or directory, unlink '/opt/build/repo/public/static/c79ed69a45dfa3eb3fa94442cc61454a/photo-1499479387933-4567e710809c.jpg' 11:17:18 AM: 11:17:18 AM: 11:17:18 AM: Error: ENOENT: no such file or directory, unlink '/opt/build/repo/public/stati c/c79ed69a45dfa3eb3fa94442cc61454a/photo-1499479387933-4567e710809c.jpg' 11:17:18 AM: 11:17:18 AM: not finished Generating image thumbnails - 13.367s 11:17:18 AM: not finished run page queries - 2.574s ... 11:17:18 AM: Failed during stage 'building site': Build script returned non-zero exit code: 2 11:17:18 AM: Finished saving boot dependencies 11:17:18 AM: Started saving go dependencies 11:17:18 AM: Finished saving go dependencies 11:17:18 AM: Build failed due to a user error: Build script returned non-zero exit code: 2 11:17:18 AM: Failing build: Failed to build site 11:17:19 AM: Finished processing build request in 47.137682029s

[marcoSven]

here is another one

11:47:09 AM: success run static queries - 0.106s - 16/16 150.38/s 11:47:13 AM: error [object Object] ENOENT: no such file or directory, unlink '/opt/build/repo/public/static/be02825633540f618f4010844e830d2f/photo-1541337326331-c1bd6156f6f6.jpg' 11:47:13 AM: 11:47:13 AM: 11:47:13 AM: Error: ENOENT: no such file or directory, unlink '/opt/build/repo/public/stati c/be02825633540f618f4010844e830d2f/photo-1541337326331-c1bd6156f6f6.jpg' 11:47:13 AM: 11:47:13 AM: not finished Generating image thumbnails - 14.152s 11:47:13 AM: not finished run page queries - 3.623s 11:47:13 AM: (sharp:1364): GLib-CRITICAL : 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed 11:47:13 AM: (sharp:1364): GLib-CRITICAL : 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed 11:47:13 AM: (sharp:1364): GLib-CRITICAL : 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed 11:47:13 AM: (sharp:1364): GLib-CRITICAL : 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed 11:47:13 AM: (sharp:1364): GLib-CRITICAL : 18:47:13.145: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed 11:47:13 AM: (sharp:1364): GLib-CRITICAL : 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed 11:47:16 AM: bash: line 1: 1364 Segmentation fault (core dumped) gatsby build --log-pages --verbose 11:47:16 AM: ​ 11:47:16 AM: ┌─────────────────────────────┐ 11:47:16 AM: │ "build.command" failed │ 11:47:16 AM: └─────────────────────────────┘ ... 11:47:16 AM: Finished saving go dependencies 11:47:16 AM: Build failed due to a user error: Build script returned non-zero exit code: 2 11:47:16 AM: Failing build: Failed to build site 11:47:16 AM: Failed during stage 'building site': Build script returned non-zero exit code: 2 11:47:16 AM: Finished processing build request in 48.689597638s

[styxlab]

The sharp errors look scary but can be ignored (there is an async issue in the gatsby-plugin-sharp plugin, but the images are still generated). The other build errors, however, should not happen and I am surprised to see those given that I also deploy on Netlify without problems. Could you please first test, if you also see those errors when building locally? Maybe it is an issue with the volume, how many images are you processing?

[marcoSven]

Funny I just had the first time the same error on my local.

1. yarn clean
2. yarn build

I looked for the file in demo/public/static and it did exist. I ran yarn build and everything was fine 🙃. Any advice on what to look for? I am just starting out with Gatsby,

wondering if

• this is related to the async issue in the gatsby-plugin-sharp, though I do not see the error every time the build fails.
• memory issue

how many images are you processing?

3.406837832 Number of node types: 16. Nodes per type: SitePage: 41, SitePlugin: 64, Site: 1, SiteBuildMetadata: 1, Directory: 3, File: 32, ImageSharp: 28, GhostTag: 14, GhostSettings: 1, GhostAuthor: 1, GhostPage: 4, HtmlRehype: 15, GhostPost: 11, ghostConfig: 1, ContactPage: 1, TopicsPage: 1

[marcoSven]

So I signed up for Gatsby, I made 3 deploys so far, and they always succeded in Gatsby but failed on Netlify.

I experienced the issues once more on my local, so I am not 100% confident that building on Gatsby solves my issue.

[styxlab]

I also build on Gatsby Cloud and not directly on Netlify, but I am also not seeing the issue locally. Could you please provide your repo with (temporary) API keys, so I can test your site locally on my machine? This is most likely either a problem with the used Gatsby version or related to your (local/Netlify) build system.

Apart from clearing the cache with yarn clean, did you also

1. remove your public folder before building?
2. remove node_modules + yarn.lock, then issue yarn to recreate?

The error messages points to dangling cache files.

[marcoSven]

I also build on Gatsby Cloud and not directly on Netlify

Not sure if I like that dependency 😄 but it works flawlessly so far. I am currently building simultaneously on both and Netlify always fails. I have removed the lock file btw and the build installs the packages with npm. I use yarn locally.

remove your public folder before building? remove node_modules + yarn.lock, then issue yarn to recreate?

I tried a lot of things replicated the same steps after a fail, but since the build time takes a while this is a painful time suck. So far I have not been able to consistently reproduce the issue on my local. The public folder is removed when you run clean, no?

This is most likely either a problem with the used Gatsby version or related to your (local/Netlify) build system.

Local - Gatsby CLI version: 2.12.107 node v12.19.0 Netlify - gatsby-cli@2.12.113 node v12.18.0

Gatsby - Gatsby CLI version: 2.12.113 Node v12.19.0

I still think it's related to image processing and running out of memory, and considering the lighthouse results mentioned in #203 images not correctly sized etc maybe improvements will help with this issue.

Error: ENOENT: no such file or directory, chmod '/opt/build/repo/public/static

Thoughts on how I can trace the error? All I was able to confirm is that the file does exist at least after the failed build, not sure where the error is thrown and how to debug.

[styxlab]

I'd be surprised if the error is related to image processing and running out of memory. You can turn off Gatsby image processing entirely by:

• removing gatsby-rehype-inline-images (for inline images)
• setting the flag gatsbyImages to false in mediaConfig.js (for feature images).

This could be a first step in trying to localize the issue.

[marcoSven]

I already removed gatsby-rehype-inline-images and I will give removing the feature images a try

[marcoSven]

While I looked at #202 I cloned this repository changed content API settings in /dev/gatsby-theme-try-ghost/packages/gatsby-theme-try-ghost/src/utils/.ghost.json

installed packages with yarn and ran yarn build

and got this error Error: ENOENT: no such file or directory, chmod '/dev/gatsby-theme-try-ghost/demo/public/static/d5aa518ea784e1b659 b6bf9a69bd6f76/site-meta.png'

yarn build right after no error

[styxlab]

and got this error Error: ENOENT: no such file or directory, chmod '/dev/gatsby-theme-try-ghost/demo/public/static/d5aa518ea784e1b659 b6bf9a69bd6f76/site-meta.png'

Sounds like a duplicate of #188 if you are building on macOS. I think this error is different from the ones showing the unlink error message, although they could have the same root cause. My hunch is that Gatsby has a problem with with cache invalidation. If we can fully replicate the issues we can report them upstream. Before, I will update the project to the latest gatsby version, so we don't report intermittent problems.

[marcoSven]

If we can fully replicate the issues we can report them upstream

Try https://gitpod.io/#snapshot/142e3a5b-3c8c-46c5-ad99-6fbaa85836e7

run yarn develop

[styxlab]

After clearing the cache, I can see the issue now. Actually, I think it is the same as gatsby/issues/27984.

[styxlab]

Fun fact: this error tells you something about the speed of your file-system. The faster your file writes the less likely you run into this error. Final fix is on its way...

[styxlab]

@marcoSven: Could you please verify that the reported build errors are fixed?

[marcoSven]

I had a hunch something hardware is related to this intermittent issue. Tells you something about Netlify vs Gatsby Cloud - kind of annoyed this was frequently happening on my Mac ;)

Looks like it is fixed!!! 🎉

[styxlab]

Perfect... and thanks for bringing this up. Both this project and official Gatsby benefited from it.