Generating image thumbnails count keeps incrementing

[tol-is]

I noticed on my terminal that the Generating image thumbnails progress was taking too long and the count showing an unusually large number.

That was strange, because I only have 2 images in my project so far, so 400+ thumbnails are too many.

I noticed, that every time I save a file, the thumbnails count keeps incrementing. Look at the screenshot at the top its about 168 thumbs, at the bottom 448. No images were added in between.

I'm not sure, how to reproduce, it looks like something happens with the .chace, and in fact as soon as deleted the .cache and restarted the project, it's all working again as expected.

Environment

System: OS: macOS 10.14 CPU: (8) x64 Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz Shell: 3.2.57 - /bin/bash Binaries: Node: 11.1.0 - ~/.nvm/versions/node/v11.1.0/bin/node Yarn: 1.12.3 - /usr/local/bin/yarn npm: 6.4.1 - ~/.nvm/versions/node/v11.1.0/bin/npm Browsers: Chrome: 71.0.3578.98 Firefox: 62.0.3 Safari: 12.0 npmPackages: gatsby: ^2.0.53 => 2.0.73 gatsby-image: ^2.0.20 => 2.0.25 gatsby-plugin-eslint: ^2.0.1 => 2.0.1 gatsby-plugin-manifest: ^2.0.9 => 2.0.12 gatsby-plugin-offline: ^2.0.16 => 2.0.20 gatsby-plugin-postcss: ^2.0.2 => 2.0.2 gatsby-plugin-react-helmet: ^3.0.2 => 3.0.4 gatsby-plugin-sharp: ^2.0.14 => 2.0.15 gatsby-plugin-styled-components: ^3.0.4 => 3.0.4 gatsby-source-filesystem: ^2.0.8 => 2.0.12 gatsby-transformer-sharp: ^2.1.8 => 2.1.9 npmGlobalPackages: gatsby-cli: 2.4.7

[hagnerd]

Sometimes the cache can cause funky stuff to happen. If your issue is fixed now after deleting the cache I think it's safe to close the issue, unless you're able to re-create the conditions.

[eghan]

I'm having this issue as well, clearing cache does work but is less than optimal for every dev build. The problem definitely started when I added a component with a static query that fetches a large number of images... curious if I've done something wrong or if this behavior is expected? is there a way to avoid this? full source --> https://github.com/eghan/gatsby-storefront/blob/master/src/components/tag-preview.js

System: OS: macOS Sierra 10.12.2 CPU: (8) x64 Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz Shell: 3.2.57 - /bin/bash Binaries: Node: 11.0.0 - /usr/local/bin/node Yarn: 1.10.1 - /usr/local/bin/yarn npm: 6.4.1 - /usr/local/bin/npm Languages: Python: 2.7.10 - /usr/bin/python Browsers: Chrome: 72.0.3626.121 Safari: 10.0.2 npmPackages: gatsby: ^2.1.17 => 2.1.17 gatsby-image: ^2.0.20 => 2.0.20 gatsby-plugin-manifest: ^2.0.9 => 2.0.9 gatsby-plugin-node-fields: ^1.0.0 => 1.0.0 gatsby-plugin-offline: ^2.0.14 => 2.0.14 gatsby-plugin-prefetch-google-fonts: ^1.0.11 => 1.0.11 gatsby-plugin-react-helmet: ^3.0.2 => 3.0.2 gatsby-plugin-sharp: ^2.0.17 => 2.0.17 gatsby-plugin-styled-components: ^3.0.1 => 3.0.1 gatsby-plugin-typography: ^2.2.1 => 2.2.1 gatsby-source-airtable: ^2.0.2 => 2.0.2 gatsby-source-filesystem: ^2.0.20 => 2.0.20 gatsby-transformer-csv: ^2.0.4 => 2.0.4 gatsby-transformer-remark: ^2.1.12 => 2.1.12 gatsby-transformer-sharp: ^2.1.8 => 2.1.8 npmGlobalPackages: gatsby-cli: 2.4.11

[iFarmGolems]

I'd also like to know how to deal with this effectively

[pierrenel]

Having the same issue - via the gatsby shopify source plugin 🤔

[alin-faur]

I noticed the same thing after I added a new image in /image folder that is referenced in gatsby-config.js with the: { resolve: gatsby-source-filesystem, options: { name: imagesSrc, path: ${__dirname}/src/images/, }
}

[skylarweaver]

Update: Upgrading my Gatsby packages to the following version resolved this issue: "gatsby": "^2.4.1", "gatsby-background-image": "^0.5.2", "gatsby-image": "^2.0.41"

---

This issue is not resolved, even after upgrading to the latest Gatsby version. Does anyone have any advice?

I have 6 photos in src/img, and after only a couple file saves, the Generating image thumbnails loader increases to 126 thumbnails: Each consecutive file save increases that number until my machine overheats and I'm forced to clear the .cache and restart the server.

[wokejacqueline]

Why was this issue closed? AFAIK, it is still unresolved... I am also running into this problem.

My site has about 300 images. At most, I would expect 300 x 5 (so, 1500~ images.)

Gatsby-image is generating 55,000 images.

This number varies between builds, last was 53,000, and sometimes it’ll still succeed. However, now it is causing frequent segfaults when building on Now’s servers and sometimes on my local machine. And it is straight murdering my macbook everytime, whether it succeeds or fails. 🥺

[Anahkiasen]

Same issue here too, this should be reopened

[james-willett]

Why is this closed again? Still having this issue. If it's closed... at least tell us why (and what the resolution is)?

[exposuredesign]

I get the same issue - the same site build can be between 200 images and 3000+ - doesn't always fix with a cache clear. Going to spend some time diagnosing then I'll post my specs etc

[FraserThompson]

Also getting the same issue. I have a very large site with around 11,000 images which should be generating 4 JPGs and 4 WebPs per image.. My build sits on "Generating image thumbnails" for hours and the counter exceeds its limit eg. currently its at 5238/1496 thumbnails, but I can see that it's not actually doing anything because my public directory isn't growing.

Deleting the cache folder and building starts the process again, it generates a few thousand thumbnails and then sits around doing nothing while the number goes up.

I am wondering if after enough iterations of deleting the cache folder and building the build will complete...

[pecata83]

Same issue here.

I have a site using about 30 images that generate more than 1k thumbnails.

[SimonKlausLudwig]

Same issue here. Again closed why ?

[nejurgis]

Same here, around 20 images giving me 690 thumbnails, i suspect its gatsby-transformer-sharp creating all these different variations of image sizes, maybe an idea to decrease the amount of thumbnails is to decrease the maxWidth: 2048 (set in the plugin settings in gatsby-config.js )?

[arhoy]

I have a site with Gatsby with 1000 product images, but almost 8000 image thumbnails!!!! It takes 15 minutes for the development bundle to build. What the heck is going on?!

[alexfromesper]

I have a site with Gatsby with 1000 product images, but almost 8000 image thumbnails!!!! It takes 15 minutes for the development bundle to build. What the heck is going on?!

Every image that you are transforming into a fluid reset will output 4 different image files (for different screen sizes).

1000 product images = 4000 product images for just the product preview images.

Im assuming that you have individual pages with product images on those as well which are likely different sizes than the preview, so an additional 4000 images are created for those.

4000 + 4000 = 8000 images produced from using gatsby's fluid image processing.

[arhoy]

Yes that is true! Wow, crazy. How can I build up the page build speed in Gatsby then ? I am sure people have way bigger stores than mine with 100X more images... What do they do?

[alexfromesper]

Yes that is true! Wow, crazy. How can I build up the page build speed in Gatsby then ? I am sure people have way bigger stores than mine with 100X more images... What do they do?

The solution to this problem is called "distributed builds". What this does is send off the task of building all of the images to a cloud server to handle the processing rather than your main server/computer.

This feature is natively built in to gatsby's new cloud hosting platform: https://gatsbyjs.com

The hosting platform is free for the most part, and builds are insanely fast.

If you want to implement this on netlify, it will require some manual work and will cost you for each time you build. Here is a guide on how to implement it on netlify: https://www.netlify.com/blog/2020/02/25/gatsby-build-speed-improvements-with-parallel-image-processing/

[arhoy]

So I have built it on Netlify, the build time was about 11 minutes with 350 products so I can imagine it is a lot more now. My issue is why is running gatsby-develop so slow. That has nothing to do with hosting no?

[alexfromesper]

So I have built it on Netlify, the build time was about 11 minutes with 350 products so I can imagine it is a lot more now. My issue is why is running gatsby-develop so slow. That has nothing to do with hosting no?

Sadly, I haven't found a way to make development environment (gatsby develop) faster.

I recommend switching to gatsby cloud hosting though and connecting your gatsby cloud project to netlify. You will see incredible improvements in your build times. I have a project with over 9000 images and it builds the project in less then 2 minutes.

[rori4]

Why is this ignored? @KyleAMathews @pieh (tagging you both as biggest contributors to the gatsby project) There are numerous issues around the gatsby-plugin-sharp plugin. It generates tons of thumbnails which slow build time even when developing and there is no way to skip it.

Probably related: https://github.com/gatsbyjs/gatsby/issues/20816

[hvitis]

+1 I am having the same problems

1:37:40 AM: success Generating image thumbnails - 807.589s - 1186/1186 1.47/s 1:37:40 AM: Generated public/sw.js, which will precache 20 files, totaling 749281 bytes. 1:37:40 AM: success onPostBuild - 0.174s 1:37:40 AM: info Done building in 820.666123289 sec

over 1200 images on blog with 20 posts (around 40 images)

[alex-troyer]

+1 I am having the same problems

1:37:40 AM: success Generating image thumbnails - 807.589s - 1186/1186 1.47/s 1:37:40 AM: Generated public/sw.js, which will precache 20 files, totaling 749281 bytes. 1:37:40 AM: success onPostBuild - 0.174s 1:37:40 AM: info Done building in 820.666123289 sec

over 1200 images on blog with 20 posts (around 40 images)

Each time you generate a fluid image, you are generating images for different screen sizes (I believe 3 or 4 different sizes). This means that you are generating a minimum 3 images for 1 use case. If you have a post preview image for the post cards and a featured image on the post, that is already a minimum of 6 images generated.

This isnt necessarily a problem with gatsby's image processing methods, its an inherit problem with generating optimized images for all screen sizes. If you want to generate less images, use a fixed query instead of fluid.

[alex-troyer]

Also for those in the future who are looking for quicker build times, look into parallel builds. Gatsby's hosting service offers this in all of their hosting plans.

For local development, avoid clearing the build cache.

[hvitis]

solved it by using

• fixed query instead of fluid.
• resizing images
• changing gatsby-sharp plugin

Now the build is done in 200s instead of 15m

Thanks!