search

gatsbyjs / gatsby

The best React-based framework with performance, scalability and security built in.
https://www.gatsbyjs.com
MIT License
55.27k stars 10.31k forks source link

Errors when processing large(ish) numbers of images with gatsby-plugin-sharp #6291

Closed jp887 closed 4 years ago

jp887 commented 6 years ago

Description

I'm processing around 1500 images with gatsby-plugin-sharp/gatsby-transformer-sharp and GraphQL, to display with gatsby-image as per this page in the docs.

For smaller numbers of images I have had no issues, but now that I'm processing 1500 images the build randomly fails with the following error:

(sharp:104): GLib-GObject-WARNING **: 09:35:11.293: gtype.c:4265: type id '0' is invalid

(sharp:104): GLib-GObject-WARNING **: 09:35:11.293: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:104): GLib-GObject-WARNING **: 09:35:11.293: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation

(sharp:104): GLib-GObject-CRITICAL **: 09:35:11.293: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed
Received 'segmentation fault' signal

This happens during the "Generating image thumbnails" step in the build process and it occurs apparently randomly (sometimes 10% of the way through the images, sometimes at 80%, sometimes not at all). Therefore I do not believe it is caused by a "faulty image".

jlin95 commented 5 years ago

@grantglidewell Did you put const sharp = require('sharp') sharp.simd(false) sharp.cache(false)

this in any of the API hooks in gatsby-node.js or did you just place this block at the top of the file?

grantglidewell commented 5 years ago

just in the file, not in any exports. this was per @DSchau

jlin95 commented 5 years ago

just in the file, not in any exports. this was per @DSchau

I see. I tried that in my project. So while the build passed, some image thumbnails were still not getting generated. Wondering if there were other things you had to implement in the code in order for this to work. We don't use Netlify for our builds :)

Thanks!

jamessimone commented 5 years ago

@TylerBarnes your comment is very interesting - while I am not experiencing the issue at hand, I am experiencing massive slowdowns in builds caused by gatsby-remark-image / filesystem constantly rebuilding all of my image files on deploy. I made an example repo and filed an issue with the creator of the Netlify cache plugin, but he hasn’t responded as of yet and I’m going half mad trying to figure out where I’ve gone wrong with my caching strategy. Any insight / tips on correctly caching images downloaded using createRemoteFileNode (TL;DR from my linked example repo, I try to cache the created gatsby node containing the images), that would be EXTREMELY appreciated

MaximeHeckel commented 5 years ago

Hey, I had the exact same issue recently with gatsby-images and gatsby-plugin-sharp. I'd get the same error messages as @grantglidewell on Netlify, and not all my images would be generated at build time (which resulted in only 2 out of 5 images being able to render on my website, for the rest I'd only get the blurred version) After 2 hours of trial and error I tried to update to the latest "gatsby-plugin-sharp": "^2.2.10", (as of the day this comment is posted) and my builds are now 100% successful and the images are properly processed. Hope that helps Note I'm on "gatsby": "^2.13.52",

dmaynard commented 5 years ago

Thank you @MaximeHeckel I was having the same problem with local builds. gatsby build seemed to exit before all of the images were packaged (but gatsby develop worked fine). Upgrading to "gatsby-plugin-sharp": "^2.2.10" seems to have fixed this problem for me.

antoinerousseau commented 5 years ago

I'm having the same problem with a local gatsby build if I don't disable shard's simd and cache.

[===========                 ]   5.893 s 344/850 40% Generating image thumbnails
/bin/sh: line 1:  5438 Segmentation fault: 11  gatsby build
error Command failed with exit code 139.

gatsby@2.13.73 gatsby-plugin-sharp@2.2.13 gatsby-transformer-sharp@2.2.7

node v10.15.3 on macOS

kpennell commented 5 years ago

Downgrading to gatsby-plugin-sharp@2.2.14 fixes the issue.

per: https://github.com/gatsbyjs/gatsby/issues/16957#issuecomment-523981633

kuworking commented 4 years ago

In my case this has happened in netlify but not in local, with 1724 images found in the public folder

Previous build run ok and this one has added ~20 new images (to be translated into 100-200? in the public folder)

"gatsby-plugin-sharp": "^2.2.12", "gatsby-remark-images": "^3.1.12", "gatsby-transformer-sharp": "^2.2.6",

10:01:32 AM: Build ready to start
10:01:52 AM: build-image version: 9e0f207a27642d0115b1ca97cd5e8cebbe492f63
10:01:52 AM: build-image tag: v3.3.2
10:01:52 AM: buildbot version: a2bab682ae0761baa4d123ebd00c57be933655dc
10:01:52 AM: Fetching cached dependencies
10:01:52 AM: Starting to download cache of 254.9KB
10:01:52 AM: Finished downloading cache in 79.77385ms
10:01:52 AM: Starting to extract cache
10:01:52 AM: Failed to fetch cache, continuing with build
10:01:52 AM: Starting to prepare the repo for build
10:01:52 AM: No cached dependencies found. Cloning fresh repo
10:01:52 AM: git clone https://github.com/xxx/xxx
10:01:56 AM: Preparing Git Reference refs/heads/netlify
10:01:57 AM: Found netlify.toml. Overriding site configuration
10:01:57 AM: Starting build script
10:01:57 AM: Installing dependencies
10:01:58 AM: Downloading and installing node v10.17.0...
10:01:58 AM: Downloading https://nodejs.org/dist/v10.17.0/node-v10.17.0-linux-x64.tar.xz...
10:01:59 AM: 
#######################################
10:01:59 AM:                               54.7%
10:01:59 AM: 
########################
10:01:59 AM: ################################################ 100.0%
10:01:59 AM: Computing checksum with sha256sum
10:01:59 AM: Checksums matched!
10:02:01 AM: Now using node v10.17.0 (npm v6.11.3)
10:02:01 AM: Attempting ruby version 2.6.2, read from environment
10:02:02 AM: Using ruby version 2.6.2
10:02:03 AM: Using PHP version 5.6
10:02:03 AM: Started restoring cached node modules
10:02:03 AM: Finished restoring cached node modules
10:02:03 AM: Installing NPM modules using NPM version 6.11.3
10:02:05 AM: npm
10:02:05 AM: WARN deprecated core-js@2.6.10: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
10:02:08 AM: npm
10:02:08 AM: WARN deprecated fsevents@1.2.9: One of your dependencies needs to upgrade to fsevents v2: 1) Proper nodejs v10+ support 2) No more fetching binaries from AWS, smaller package size
10:02:10 AM: npm WARN deprecated core-js@1.2.7: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
10:02:25 AM: npm WARN
10:02:25 AM:  deprecated @types/vfile-message@2.0.0: This is a stub types definition. vfile-message provides its own type definitions, so you do not need this installed.
10:03:03 AM: > sharp@0.23.2 install /opt/build/repo/node_modules/sharp
10:03:03 AM: > (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
10:03:03 AM: info
10:03:03 AM: sharp
10:03:03 AM:  Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz
10:03:05 AM: > core-js@2.6.10 postinstall /opt/build/repo/node_modules/core-js
10:03:05 AM: > node postinstall || echo "ignore"
10:03:05 AM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
10:03:05 AM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
10:03:05 AM: > https://opencollective.com/core-js 
10:03:05 AM: > https://www.patreon.com/zloirock 
10:03:05 AM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
10:03:06 AM: > styled-components@4.3.2 postinstall /opt/build/repo/node_modules/styled-components
10:03:06 AM: > node ./scripts/postinstall.js || exit 0
10:03:06 AM: Use styled-components at work? Consider supporting our development efforts at https://opencollective.com/styled-components
10:03:06 AM: > gatsby-telemetry@1.1.37 postinstall /opt/build/repo/node_modules/gatsby-telemetry
10:03:06 AM: > node src/postinstall.js
10:03:06 AM: > cwebp-bin@5.1.0 postinstall /opt/build/repo/node_modules/cwebp-bin
10:03:06 AM: > node lib/install.js
10:03:07 AM:   ✔ cwebp pre-build test passed successfully
10:03:07 AM: > mozjpeg@6.0.1 postinstall /opt/build/repo/node_modules/mozjpeg
10:03:07 AM: > node lib/install.js
10:03:07 AM:   ✔ mozjpeg pre-build test passed successfully
10:03:07 AM: > pngquant-bin@5.0.2 postinstall /opt/build/repo/node_modules/pngquant-bin
10:03:07 AM: > node lib/install.js
10:03:07 AM:   ✔ pngquant pre-build test passed successfully
10:03:07 AM: > gatsby-cli@2.8.11 postinstall /opt/build/repo/node_modules/gatsby/node_modules/gatsby-cli
10:03:07 AM: > node scripts/postinstall.js
10:03:08 AM: > gatsby@2.17.15 postinstall /opt/build/repo/node_modules/gatsby
10:03:08 AM: > node scripts/postinstall.js
10:03:10 AM: npm
10:03:10 AM: notice created a lockfile as package-lock.json. You should commit this file.
10:03:10 AM: npm
10:03:10 AM: WARN tsutils@3.17.1 requires a peer of typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta but none is installed. You must install peer dependencies yourself.
10:03:10 AM: npm WARN
10:03:10 AM:  optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
10:03:10 AM: npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
10:03:10 AM: npm
10:03:10 AM: WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.2 (node_modules/xxx/node_modules/fsevents):
10:03:10 AM: npm WARN notsup
10:03:10 AM:  SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
10:03:10 AM: npm
10:03:10 AM:  WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.2 (node_modules/chokidar/node_modules/fsevents):
10:03:10 AM: npm WARN
10:03:10 AM: notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
10:03:10 AM: added 2219 packages from 1064 contributors and audited 29407 packages in 66.353s
10:03:10 AM: found 0 vulnerabilities
10:03:10 AM: NPM modules installed
10:03:10 AM: Started restoring cached go cache
10:03:10 AM: Finished restoring cached go cache
10:03:10 AM: unset GOOS;
10:03:10 AM: unset GOARCH;
10:03:10 AM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
10:03:10 AM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
10:03:10 AM: go version >&2;
10:03:10 AM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
10:03:10 AM: go version go1.12 linux/amd64
10:03:10 AM: Installing missing commands
10:03:10 AM: Verify run directory
10:03:10 AM: Executing user command: gatsby build
10:03:13 AM: success open and validate gatsby-configs - 0.092s
10:03:14 AM: success load plugins - 1.257s
10:03:14 AM: success onPreInit - 0.009s
10:03:14 AM: success delete html and css files from previous builds - 0.011s
10:03:14 AM: success initialize cache - 0.008s
10:03:14 AM: success copy gatsby files - 0.048s
10:03:14 AM: success onPreBootstrap - 0.011s
10:03:22 AM: success source and transform nodes - 7.410s
10:03:22 AM: success building schema - 0.431s
10:03:22 AM: success createPages - 0.197s
10:03:22 AM: success createPagesStatefully - 0.115s
10:03:23 AM: success onPreExtractQueries - 0.004s
10:03:23 AM: success update schema - 0.050s
10:03:23 AM: success extract queries from components - 0.429s
10:03:23 AM: success write out requires - 0.006s
10:03:23 AM: success write out redirect data - 0.001s
10:03:24 AM: success Build manifest and related icons - 1.128s
10:03:24 AM: success onPostBootstrap - 1.154s
10:03:24 AM: ⠀
10:03:24 AM: info bootstrap finished - 13.927 s
10:03:24 AM: ⠀
10:04:01 AM: success Building production JavaScript and CSS bundles - 33.009s
10:04:01 AM: success Rewriting compilation hashes - 0.001s
10:05:13 AM: error error copying file from /opt/build/repo/content/recipes/images/none.jpg to /opt/build/repo/public/static/none-49bb16de8c78d925cf6d9aa24b8c556c.jpg { [Error: ENOENT: no such file or directory, unlink '/opt/build/repo/public/static/none-49bb16de8c78d925cf6d9aa24b8c556c.jpg']
10:05:13 AM:   errno: -2,
10:05:13 AM:   code: 'ENOENT',
10:05:13 AM:   syscall: 'unlink',
10:05:13 AM:   path:
10:05:13 AM:    '/opt/build/repo/public/static/none-49bb16de8c78d925cf6d9aa24b8c556c.jpg' }
10:05:22 AM: *** Error in `node': double free or corruption (fasttop): 0x00007fdd1c1727a0 ***
10:05:22 AM: ======= Backtrace: =========
10:05:22 AM: /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fdd4940b7e5]
10:05:22 AM: /lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fdd4941437a]
10:05:22 AM: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fdd4941853c]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(+0x208f49)[0x7fdd23a44f49]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libglib-2.0.so.0(g_hash_table_replace+0x1f0)[0x7fdd402abfd0]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(+0x208ec0)[0x7fdd23a44ec0]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(vips_image_set+0x25)[0x7fdd23a45665]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(vips_image_set_string+0x50)[0x7fdd23a46890]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(vips__exif_parse+0x2be)[0x7fdd239e7e7e]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(vips_image_set+0x50)[0x7fdd23a45690]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(vips_image_set_blob+0x60)[0x7fdd23a461a0]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(vips__exif_update+0x1b5)[0x7fdd239e8095]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(+0x1cbe60)[0x7fdd23a07e60]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(vips__jpeg_write_file+0x1d1)[0x7fdd23a08481]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(+0x1cebea)[0x7fdd23a0abea]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(vips_object_build+0x19)[0x7fdd23a36b29]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42(vips_cache_operation_buildp+0x48)[0x7fdd23a41f98]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips-cpp.so.42(_ZN4vips6VImage18call_option_stringEPKcS2_PNS_7VOptionE+0x65)[0x7fdd405d3905]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/../../vendor/lib/libvips-cpp.so.42(_ZNK4vips6VImage8jpegsaveEPKcPNS_7VOptionE+0x59)[0x7fdd405ddf99]
10:05:22 AM: /opt/build/repo/node_modules/sharp/build/Release/sharp.node(_ZN14PipelineWorker7ExecuteEv+0x3166)[0x7fdd48180556]
10:05:22 AM: node[0xa6ab3e]
10:05:22 AM: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fdd497656ba]
10:05:22 AM: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fdd4949b41d]
10:05:22 AM: ======= Memory map: ========
10:05:22 AM: 00400000-0243e000 r-xp 00000000 08:01 5623487                            /opt/buildhome/.nvm/versions/node/v10.17.0/bin/node
10:05:22 AM: 0263d000-0263f000 r--p 0203d000 08:01 5623487                            /opt/buildhome/.nvm/versions/node/v10.17.0/bin/node
10:05:22 AM: 0263f000-0264b000 rw-p 0203f000 08:01 5623487                            /opt/buildhome/.nvm/versions/node/v10.17.0/bin/node
10:05:22 AM: 0264b000-02666000 rw-p 00000000 00:00 0
10:05:22 AM: 044fc000-07e3a000 rw-p 00000000 00:00 0                                  [heap]
10:05:22 AM: 434880000-434900000 rw-p 00000000 00:00 0
10:05:22 AM: 3138d80000-3138e00000 rw-p 00000000 00:00 0
10:05:22 AM: 3d10300000-3d10380000 rw-p 00000000 00:00 0
(.......... more like these)
10:05:22 AM: 20c8ca180000-20c8ca200000 rw-p 00000000 00:00 0
10:05:22 AM: 20e266280000-20e266300000 rw-p 00000000 00:00 0
10:05:30 AM: failed during stage 'building site': Build script returned non-zero exit code: 134
10:05:22 AM: 20e801900000-20e801980000 rw-p 00000000 00:00 0
10:05:22 AM: 20f21a300000-20f21a380000 rw-p 00000000 00:00 0
10:05:22 AM: 20f398800000-20f398880000 rw-p 00000000 00:00 0
10:05:30 AM: Shutting down logging, 960 messages pending
10:05:22 AM: 21302f780000-21302f800000 rw-p 00000000 00:00 0
10:05:22 AM: 213cad700000-213cad780000 rw-p 00000000 00:00 0
10:05:22 AM: 214646c00000-214646c80000 rw-p 00000000 00:00 0
kuworking commented 4 years ago

After having updated to 

    "gatsby-plugin-sharp": "^2.3.0",
    "gatsby-transformer-sharp": "^2.3.5",

the error is then

11:06:15 AM: error vips_image_get: field "exif-data" not found
11:06:15 AM: UNHANDLED REJECTION vips_image_get: field "exif-data" not found
11:06:15 AM: See our docs page for more info on this error: https://gatsby.dev/issue-how-to
11:06:15 AM: 
11:06:15 AM:   Error: vips_image_get: field "exif-data" not found
11:06:15 AM: 
11:06:15 AM: Skipping functions preparation step: no functions directory set
11:06:15 AM: Caching artifacts
11:06:15 AM: Started saving node modules
11:06:15 AM: Finished saving node modules
11:06:15 AM: Started saving pip cache
11:06:15 AM: Finished saving pip cache
11:06:15 AM: Started saving emacs cask dependencies
11:06:15 AM: Finished saving emacs cask dependencies
11:06:15 AM: Started saving maven dependencies
11:06:15 AM: Finished saving maven dependencies
11:06:15 AM: Started saving boot dependencies
11:06:15 AM: Finished saving boot dependencies
11:06:15 AM: Started saving go dependencies
11:06:15 AM: Finished saving go dependencies
11:06:15 AM: Error running command: Build script returned non-zero exit code: 1
11:06:15 AM: failed during stage 'building site': Build script returned non-zero exit code: 1
11:06:15 AM: Failing build: Failed to build site
11:06:15 AM: Finished processing build request in 5m3.761363935s

then I've added to gatsby-node

const sharp = require('sharp')
sharp.cache(false)
sharp.simd(false)

And now netlify doesn't finish (it is building like forever*), and in local it doesn't build (node v10.16.0)

*it's finished: 11:59:16 AM: /usr/local/bin/build: line 60: 1345 Segmentation fault (core dumped) gatsby build

Right now I'm not able to deploy my site, and I haven't introduced any change apart from new content 😥

success run page queries - 54.340 s — 108/108 1.99 queries/second
⠙ run page queries — 108/108 1.99 queries/second
 [=-----------------------------] 3158/71874 73.9 s 4% Generating image thumbnails
(sharp:6564): GLib-GObject-WARNING **: 11:51:59.486: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.527: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.527: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.527: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.528: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.534: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.535: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.535: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.537: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.538: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.538: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.538: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.539: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.539: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.539: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.540: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.541: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.541: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.541: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.542: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.542: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.543: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.543: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.543: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.544: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.544: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.544: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.545: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.545: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.547: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.547: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.548: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.553: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.553: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.554: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.554: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.555: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.555: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:6564): GLib-GObject-WARNING **: 11:51:59.555: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation

(sharp:6564): GLib-GObject-CRITICAL **: 11:51:59.556: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed
 [====--------------------------] 10564/71874 130.6 s 14% Generating image thumbnailsException code=0xc0000005 flags=0x0 at 0x00007FF84D575CB5. Access violation - attempting to write data at address 0x0000000000000000
error Command failed with exit code 3221225477.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 3221225477
Command: C:\Program Files\nodejs\node.exe
Arguments: C:\Program Files (x86)\Yarn\lib\cli.js build
Directory: D:\CODING\xxx\site
Output:

After cleaning, I get this consistent error (in local)

 [==----------------------------] 3492/64296 70.7 s 5% Generating image thumbnails
(sharp:20552): GLib-GObject-CRITICAL **: 12:01:47.412: g_value_unset: assertion 'G_IS_VALUE (value)' failed
Exception code=0xc0000005 flags=0x0 at 0x000000005F305997. Access violation - attempting to read data at address 0x0000000000000018

Any idea?

niklasravnsborg commented 4 years ago

At my company we also had problems with this for a very long time. Over a longer period of time I tried Netlify, Zeit Now and buddy.works. Time to time we hit resource limits or this error occurred. We recently made the switch to GitHub Actions and had no problems so far. Probably because the execution runtime is really generous with RAM and CPU resources.

kuworking commented 4 years ago

@niklasravnsborg but this is happening in local as well ... shouldn't be a problem (it is a really simple website mine)

antoinerousseau commented 4 years ago

Check your yarn.lock to see if the versions of sharp are the same

kuworking commented 4 years ago

I've just deleted every node_modules and deleted yarn.lock and run yarn again, but build in local has failed again, now with an exit code = 'Exit code: 3221226356' without any more information

I'm going to move out all images and see if it runs again, and then I guess the solution would be to put all images in static (?)

antoinerousseau commented 4 years ago

Ok but can you actually check the used shard versions in yarn.lock? (dependencies of dependencies) Some people experience issues when there are different shard versions being used at the same time.

kuworking commented 4 years ago

*You mean sharp I guess

I see a difference in gatsby-plugin-manifest

gatsby-plugin-manifest@2.2.13:
    sharp "^0.23.0"
gatsby-plugin-sharp@^2.3.0:
    sharp "^0.23.2"
gatsby-transformer-sharp@^2.3.5:
    sharp "^0.23.2"
sharp@^0.23.0, sharp@^0.23.2:
  version "0.23.2"

I've updated the plugin but local build fails with the same code 3221226356

Looking at yarn.lock (deleted and yarn again) I can see I have 2 entries of the plugin, the older one and the new one

gatsby-plugin-manifest@2.2.13:
  version "2.2.13"
  resolved "https://registry.yarnpkg.com/gatsby-plugin-manifest/-/gatsby-plugin-manifest-2.2.13.tgz#256242bd8658f19240133439dd3b0c3a6cd4107f"
  integrity sha512-oUZYdDyqacalq6VkEc7QZIqLZNwAxsBzk9UtJShM6e9otm1XY/R9DGtyS70qnWy9kDb8y5T4qhPVjIGhG1ulgA==
  dependencies:
    "@babel/runtime" "^7.5.5"
    gatsby-core-utils "^1.0.7"
    semver "^5.7.1"
    sharp "^0.23.0"

gatsby-plugin-manifest@2.2.28, gatsby-plugin-manifest@^2.2.28:
  version "2.2.28"
  resolved "https://registry.yarnpkg.com/gatsby-plugin-manifest/-/gatsby-plugin-manifest-2.2.28.tgz#17464f404c58b748ebd34efbf783a6f5b0f85f1b"
  integrity sha512-VvpVF4uE2IkNW+WXN8UvB2slU3tldR054beyvmOAt4HkKgnRLBU8j78yGrEhzdUHqbt2wgroI2KI9VYMiicG4Q==
  dependencies:
    "@babel/runtime" "^7.7.2"
    gatsby-core-utils "^1.0.19"
    semver "^5.7.1"
    sharp "^0.23.2"
antoinerousseau commented 4 years ago

Yes sorry I meant sharp ;) Your yarn.lock looks good, it only uses sharp v0.23.2... you can try npm rebuild sharp. If not, you can try setting sharp version to 0.22.1 in package.json by adding this:

"resolutions": {
    "sharp": "0.22.1"
  }

And then rebuild (or remove node_modules and reinstall)

Refs: https://github.com/gatsbyjs/gatsby/issues/16957#issuecomment-524078143 https://github.com/gatsbyjs/gatsby/issues/16985

kuworking commented 4 years ago

I've just written there, and now I'm gonna try downgrading to 0.22.1 to see if that works (thanks!)

*Without luck so far, updating here https://github.com/gatsbyjs/gatsby/issues/16985#issuecomment-554767707

muescha commented 4 years ago

on small repos no problem, then i tested with the gatsby/www:

Bildschirmfoto 2019-11-21 um 05 23 06

it shows 100% but still left some files - it stays at this point - then this error after some minutes waiting:

info bootstrap finished - 299.684 s
⠀

 ERROR 

UNHANDLED REJECTION stdout read ENOTCONN

  Error: stdout read ENOTCONN

  - child_process.js:311 createSocket
    internal/child_process.js:311:14

  - child_process.js:424 ChildProcess.spawn
    internal/child_process.js:424:23

  - index.js:204 module.exports
    [www]/[imagemin-pngquant]/[execa]/index.js:204:26

  - index.js:52 
    [www]/[imagemin-pngquant]/index.js:52:13

  - index.js:71 Function.module.exports.buffer
    [www]/[imagemin]/index.js:71:31

  - process-file.js:200 
    [www]/[gatsby-plugin-sharp]/process-file.js:200:105

  - process-file.js:179 
    [www]/[gatsby-plugin-sharp]/process-file.js:179:7

  - async Promise.all

not finished run queries - 606.197s

———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
3408 pages                                                           In Progress                                                           gatsbyjs.org
(sharp:39117): GLib-CRITICAL **: 05:24:39.020: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:39117): GLib-CRITICAL **: 05:24:39.023: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:39117): GLib-CRITICAL **: 05:24:39.023: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:39117): GLib-CRITICAL **: 05:24:39.023: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:39117): GLib-CRITICAL **: 05:24:39.024: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:39117): GLib-CRITICAL **: 05:24:39.024: g_hash_table_lookup: assertion 'hash_table != NULL' failed

(sharp:39117): GLib-CRITICAL **: 05:24:39.024: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed

(sharp:39117): GLib-CRITICAL **: 05:24:39.025: g_hash_table_lookup: assertion 'hash_table != NULL' failed
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Process finished with exit code 1

my sharp

$ yarn list --pattern sharp
yarn list v1.19.1
├─ gatsby-plugin-sharp@2.3.2
├─ gatsby-transformer-sharp@2.3.5
└─ sharp@0.23.3
esetnik commented 4 years ago

I am facing the same issue:

❯ yarn list --pattern sharp
yarn list v1.19.1
├─ gatsby-plugin-sharp@2.3.2
├─ gatsby-transformer-sharp@2.3.5
└─ sharp@0.23.3
✨  Done in 1.44s.
muescha commented 4 years ago

@esetnik you can try out this fix https://github.com/gatsbyjs/gatsby/issues/19576#issuecomment-557664165 from @wardpeet

muescha commented 4 years ago

@gatsbyjs/core maybe we can close this issue now?

pvdz commented 4 years ago

No. This can still be reproduced. We're looking into it at the moment.

pvdz commented 4 years ago

Ok, update time.

Happy to report that we managed to repro this in a somewhat controlled manor and were able to hand that repro over to the Sharp maintainer. We're expecting a few improvements in the situation.

Please beware that this issue is a conflation of a bunch of different cases.

Some of the binary-version-mismatch problems are likely to disappear next year, when node 8 LTS gets deprecated and we move to node 10 LTS. Node 10 supports things which allow libraries like Sharp to make the binary part easier. But seeing how we're not there yet and there's a large gap between then and now, it's good to see that we can improve the situation.

Additionally I'll go through our image queuing process to see whether we can improve the situation on Gatsby's side. We shouldn't have really triggered those segfaults and regardless of fixes to Sharp, Gatsby should behave better, which in turn should improve perf a bit at scale.

Thanks everyone for helping to report their workarounds!

ehannes commented 4 years ago

@pvdz any more updates to share?

pvdz commented 4 years ago

@ehannes Are you still seeing this on master? There were some upstream fixes in the library and last I read it was waiting for a fix of the binary they rely on as well. Additionally, we're deprecating node 8 which should bring more stability as well (for reasons). I will pick this up soon to see what's left.

ehannes commented 4 years ago

@pvdz I'm not using the latest master version but I am using Gatsby 2.18.17:

$ gatsby info --clipboard

  System:
    OS: Linux 5.3 Pop!_OS 18.04 LTS
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
    Shell: 4.4.20 - /bin/bash
  Languages:
    Python: 2.7.17 - /usr/bin/python
  Browsers:
    Chrome: 79.0.3945.117
    Firefox: 72.0.1
  npmPackages:
    gatsby: ^2.13.70 => 2.18.17
    gatsby-transformer-csv: ^2.1.21 => 2.1.21
    gatsby-transformer-yaml: ^2.2.18 => 2.2.20

When I run gatsby clean && gatsby build --prefix-paths I get the output described below. Running this command multiple times in a row usually gets me a successful build. It's really annoying and time consuming since the build already takes quite a time to finish.

$ yarn build
yarn run v1.21.1
$ gatsby clean && gatsby build --prefix-paths
info Deleting .cache, public
info Successfully deleted directories
success open and validate gatsby-configs - 0.049s
success load plugins - 0.957s
success onPreInit - 0.001s
success delete html and css files from previous builds - 0.008s
success initialize cache - 0.006s
success copy gatsby files - 0.028s
success onPreBootstrap - 0.008s
success createSchemaCustomization - 0.003s
success source and transform nodes - 0.443s
success building schema - 0.312s
success createPages - 0.004s
success createPagesStatefully - 0.090s
success onPreExtractQueries - 0.001s
success update schema - 0.017s
success extract queries from components - 0.258s
success write out requires - 0.004s
success write out redirect data - 0.001s
success Build manifest and related icons - 0.048s
success onPostBootstrap - 0.052s
⠀
info bootstrap finished - 4.334 s
⠀
success Building production JavaScript and CSS bundles - 7.830s
success Rewriting compilation hashes - 0.006s
success run queries - 11.542s - 16/16 1.39/s
[======================      ]   49.849 s 1571/1980 79% Generating image thumbnails

(sharp:26802): GLib-GObject-WARNING **: 08:18:02.392: gtype.c:4265: type id '0' is invalid

(sharp:26802): GLib-GObject-WARNING **: 08:18:02.392: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:26802): GLib-GObject-WARNING **: 08:18:02.392: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation

(sharp:26802): GLib-GObject-CRITICAL **: 08:18:02.392: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:26802): GLib-GObject-WARNING **: 08:18:02.392: gtype.c:4265: type id '0' is invalid

(sharp:26802): GLib-GObject-WARNING **: 08:18:02.392: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:26802): GLib-GObject-WARNING **: 08:18:02.392: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation

(sharp:26802): GLib-GObject-CRITICAL **: 08:18:02.392: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

[... a lot more of these errors...]

[============================]   71.682 s 1972/1980 100% Generating image thumbnails

(sharp:26802): GLib-GObject-WARNING **: 08:18:24.225: gtype.c:4265: type id '0' is invalid

(sharp:26802): GLib-GObject-WARNING **: 08:18:24.225: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:26802): GLib-GObject-WARNING **: 08:18:24.225: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation

(sharp:26802): GLib-GObject-CRITICAL **: 08:18:24.225: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:26802): GLib-GObject-CRITICAL **: 08:18:24.248: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed

(sharp:26802): GLib-GObject-WARNING **: 08:18:24.248: gtype.c:4265: type id '0' is invalid

(sharp:26802): GLib-GObject-WARNING **: 08:18:24.248: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:26802): GLib-GObject-WARNING **: 08:18:24.248: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation

success Generating image thumbnails - 72.118s - 1980/1980 27.46/s
success Building static HTML for pages - 4.873s - 14/14 2.87/s
info Done building in 89.375 sec
Done in 90.56s.

And I don't think I have multiple versions of sharp:

$ npm list sharp
myproject@0.1.0 /path-to-project
└─┬ mytheme@0.26.4
  ├─┬ gatsby-plugin-manifest@2.2.34
  │ └── sharp@0.23.4  deduped
  ├─┬ gatsby-plugin-sharp@2.3.10
  │ └── sharp@0.23.4  deduped
  ├─┬ gatsby-transformer-sharp@2.3.9
  │ └── sharp@0.23.4  deduped
  └── sharp@0.23.4
smoke-indica commented 4 years ago

Using Gatsby v2.18.22, getting this issue which disrupts both gatsby develop & gatsby build when using gatsby-remark-images-anywhere (repo issue)

npm list sharp
gatsby-starter-lumen@3.0.3 R:\Gatsby_Smoke\blog
+-- gatsby-plugin-manifest@2.2.37
| `-- sharp@0.23.4  deduped
+-- gatsby-plugin-sharp@2.3.13
| `-- sharp@0.23.4  deduped
+-- gatsby-transformer-sharp@2.3.12
| `-- sharp@0.23.4  deduped
`-- sharp@0.23.4
[==========================  ]   190.398 s 579/609 95% Generating image thumbnails
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.876: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.891: can't peek value table for type '<invalid>' which is not currently referenced
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.891: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation
(sharp:11244): GLib-GObject-CRITICAL **: 22:46:09.891: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.907: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.907: can't peek value table for type '<invalid>' which is not currently referenced
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.907: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation
(sharp:11244): GLib-GObject-CRITICAL **: 22:46:09.907: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.907: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.907: can't peek value table for type '<invalid>' which is not currently referenced
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.907: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation
(sharp:11244): GLib-GObject-CRITICAL **: 22:46:09.907: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.923: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.923: can't peek value table for type '<invalid>' which is not currently referenced
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.923: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation
(sharp:11244): GLib-GObject-CRITICAL **: 22:46:09.923: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.923: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.923: can't peek value table for type '<invalid>' which is not currently referenced
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.923: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation
[                            ]   514.207 s 72/3145 2% run queries
[======================      ]   113.895 s 72/91 79% Downloading remote files
[==========================  ]   190.513 s 579/610 95% Generating image thumbnails
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.923: ../../checkout/glib-2.61.0/gobject/gtype.c:4265: type id '0' is invalid
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.985: can't peek value table for type '<invalid>' which is not currently referenced
(sharp:11244): GLib-GObject-WARNING **: 22:46:09.985: ../../checkout/glib-2.61.0/gobject/gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation 
Exception code=0xc0000005 flags=0x0 at 0x000000006E17C0B7. Access violation - attempting to read data at address 0x0000000000000020
ehannes commented 4 years ago

@pvdz have you had time to pick this up? Yesterday, I had to rerun the gatsby build command like 10 times before the build succeeded, quite annoying when the build takes around 5 minutes to complete. I wasted like an hour or two just trying to build one of my gatsby sites :cry: 9 out of 10 times it crashes with GLib-errors like the one I posted above.

pvdz commented 4 years ago

@ehannes it so happens I have! :) Was testing yesterday and could still repro with the then newest version. But (in response?) there has been a new release with a new binary (-> https://github.com/lovell/sharp/issues/1986 ) so I'll be testing that today. If all goes well we should be able to upgrade and resolve some-if-not-all of these issues soon.

pvdz commented 4 years ago

Fwiw, I can repro the warning outputs in the new version ~but not the segfault. So perhaps that's an improvement.~ and also the core dump. Unfortunately this release is tagged node 10+ and we'll need about a month of transition time for people to migrate to node 10. Perhaps a special release can be made for us which says node 8+ but dunno. So it sounds like you'll have to wait a bit longer :/

ehannes commented 4 years ago

Yes, I saw your comments in the above mentioned sharp issue. Good work, sad it didn't pay off.

I don't quite understand your comment about the node version. Is that version related to this issue somehow? I'm running node v10.16.1...

pvdz commented 4 years ago

@ehannes no, I was able to repro in v10 as well. The comment is about the nodejs EOL cycle. In 2020 nodejs v8 reached end of life ("EOL"), meaning it won't get any more patches and fixes and people should move to node 10, which is the next long term support ("LTS") release version. So this month many packages are bumping their minimum required nodejs version to >=10, sharp being no exception. Gatsby will do so as well but we're taking a little longer to allow people to make the transition outside of the busy December month that it would otherwise have been. Hence, currently we couldn't bump the sharp version because then people on node8 (which we claim is the minimum version we support) would get node version errors when installing. (This can be worked around, I don't think we need to).

That said, the new sharp version should be more stable as a few bugs related to this issue were found and fixed. Just, apparently, not all of them.

wardpeet commented 4 years ago

I'm wondering if people still see this issue on gatsby-plugin-sharp@2.4.3. We have reduced memory usage of the sharp plugin itself.

pvdz commented 4 years ago

Keep in mind that the segfault was a real race condition problem that was only exposed by high pressure. It was not about memory. It is possible symptoms get reduced as we rely less on sharp for queuing. But they are also prepping a next release fixing some more bugs. I guess, basically as long as I can repro it and they can confirm it, there's at least a problem left.

wokejacqueline commented 4 years ago

Still experiencing this issue as of "gatsby-plugin-sharp": "2.4.4" (just upgraded from 2.3.3) & "gatsby-transformer-sharp": "2.3.13"

300-ish images processed into 1800 thumbnails of different sizes will usually fail once or twice (sometimes more) before succeeding.

Screen Shot 2020-01-30 at 9 42 08 PM

It always succeeds eventually. So I just "npm run build || npm run build || npm run build --verbose" for now lol. Thanks for looking into this, I have been experiencing this issue on my project since at least Nov 2019. :(

ehannes commented 4 years ago

@pvdz any updates on this issue? Still have massive problems with this...

pvdz commented 4 years ago

@ehannes A new version of Sharp was released (with new binary) that is supposed to fix some-if-not-all of these problems. Unfortunately it is released with a minimal nodejs version of 10 (due to EOL of node 8) and Gatsby can not bump the minimal to node 10 right now (but soon...). Meaning we can't bump Sharp at the moment to make use of these improvements.

Of course, you can try to install it locally. I haven't had a chance yet to verify the latest release, which is on my todo-list for this week.

ehannes commented 4 years ago

@pvdz I thought this bug was still present in the new Sharp version. You wrote this a month ago:

I was able to repro in v10 as well

pvdz commented 4 years ago

@ehannes yeah, and that was reported upstream, and I think there was a new release about a week ago (or two weeks?). I haven't had time to investigate. We (Gatsby) had a meetup in the US last week. No worky worky in the US for us EU people. Just haven't gotten around to it beyond that, will try to report back this week.

As for the node10 concern; the problem I mentioned is not so much the repro but rather that if we bump to the new Sharp version (0.24+) in Gatsby then about 5% of our users will have installation problems because they are still on Node 8 and npm/yarn will block installation due to Sharp requiring 10+.

Fwiw, in the end the problem (of this issue) a race condition in the binary used by the Sharp library. It is made more likely by high pressure in the rest of the system but it can happen at any time for anyone. If you want to help out feel free to look at lovell/sharp#1986 and help out. Gatsby should be able to cut over to Node 10 "soon", but I have no exact timeframe on what "soon" is.

pvdz commented 4 years ago

It looks like the next release of Sharp (0.25.0, not yet published) will fix these issues. It will require node 10 so Gatsby won't ship with it until Gatsby can bump the node minver to 10 LTS, but that shouldn't stop you from being able to use it.

So it looks like that will mitigate this problem. I was able to generate 30k thumbs without signs of the issue (where before the problem would generally repro between 3k and 6k images processed).

Will keep this open until Gatsby can ship the new, fixed, version. But please go ahead and test the "sharp": "lovell/sharp#yield" version of Sharp to confirm this fixes the problem for you.

wokejacqueline commented 4 years ago

Can confirm after forcing my project to use Sharp on the yield branch, I am no longer experiencing segfaults with image processing. Previously, I would experience them almost every build, multiple times.

THANK YOU SO MUCH! 🥳

ehannes commented 4 years ago

I can also confirm that the latest version of Sharp is finally solving my build problems! By the way, 0.25.0 is now published, as well as 0.25.1.

So grateful for this fix! Thanks for your hard work @pvdz!

TylerBarnes commented 4 years ago

Same here! So excited to see this fixed. Amazing job @pvdz

pvdz commented 4 years ago

Sharp has been bumped by https://github.com/gatsbyjs/gatsby/pull/22432 and that means this issue should now be closed. Please report back if you have Sharp related segfaults or see warning messages that have been reported in this thread.

This change has been published in the following packages:

For any other Sharp related problems or questions, please open a new issue. Thanks!

esetnik commented 4 years ago

I am still having occasional segfaults using gatsby-transformer-sharp@2.4.1

$ gatsby build
success open and validate gatsby-configs - 0.039s
/bin/sh: lscpu: not found
/bin/sh: lscpu: not found
success load plugins - 0.505s
success onPreInit - 0.007s
success delete html and css files from previous builds - 0.009s
success initialize cache - 0.006s
success copy gatsby files - 0.059s
success onPreBootstrap - 0.007s
/bin/sh: lscpu: not found
success createSchemaCustomization - 0.005s
success source and transform nodes - 0.103s
success building schema - 0.271s
success createPages - 0.000s
success createPagesStatefully - 0.091s
success updating schema - 0.028s
success onPreExtractQueries - 0.001s
success extract queries from components - 0.388s
success write out redirect data - 0.001s
success Build manifest and related icons - 0.320s
success onPostBootstrap - 0.322s
info bootstrap finished - 4.358s
success run static queries - 2.702s - 6/6 2.22/s
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with signal "SIGSEGV".
error Command failed.
Exit code: 1
Command: /usr/local/bin/node
Arguments: /opt/yarn-v1.22.4/lib/cli.js build
Directory: /builds/xxx
Output:
info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.
lucashfreitas commented 3 years ago

The issue is still happening with the latest version of gatsby sharp: "2.10.0".

My website has a lot of pages/images and it's seems impossible to get a build done using netlify. If I trigger the build using containers inside AWS or my local computer it's working fine, but not inside Netlify.

Anyone facing the same problem when trying to deploy/build large websites/ lots of images on netlify?

lucashfreitas commented 3 years ago

Quick update:

I've tried to deploy my websites using Gatsby Cloud, Vercel, and Netlify and all of them complain about the same error. ('segmentation fault' signal). Apparently it's related to sharp plugin memory consumption.

I have updated all plugins and gatsby to the latest version and it's really hard to get a build running at the moment. The only place that the build would run fine it's on my laptop which has 32GB of RAM.

I was wondering if gatsby it's not suitable for huge projects with a lot of pages and images?

I have already contacted the Gatsby Cloud support and will post updates here!

twhitson commented 3 years ago

Just wanted to add that I'm seeing the same thing on GitHub Actions as well. @lucashfreitas did you happen to get any answers from Gatsby Cloud support?

lucashfreitas commented 3 years ago

Hey @twhitson ,

I reached out to gatsby cloud support and they have found the problem. Some of the plugins that I had on my project were using a outdated version of sharp library, so the quickest solution is to change the following:

upgrade gatsby-plugin-sharp

upgrade gatsby-transformer-sharp

set resolution for sharp to "0.26.3"

To set the resolutions, you can add this entry to your package.json:

  "resolutions": {     "sharp": "^0.26.3"   }

Hope it helps!