gatsbyjs / gatsby

The best React-based framework with performance, scalability and security built in.
https://www.gatsbyjs.com
MIT License
55.28k stars 10.31k forks source link

CircleCI Production Build Fails with "Generating Image Thumbnails Failed" #26050

Closed rmcsharry closed 4 years ago

rmcsharry commented 4 years ago

Description

Locally 'npm run build' works fine. On CircleCI it fails with:

info bootstrap finished - 14.297 s
⠀
 [] 0/3 0.0 s 0% Generating image thumbnails [] 0/16 0.0 s 0% Generating image thumbnails [] 1/893 0.0 s 0% run queries [] 3/893 0.0 s 0% run queries [] 0/22 0.0 s 0% Generating image thumbnails [] 4/893 0.0 s 0% run queries [] 10/22 1.8 s 45% Generating image thumbnails [] 16/22 2.7 s 72% Generating image thumbnails [] 22/22 3.7 s 100% Generating image thumbnailsfailed Building production JavaScript and CSS bundles - 42.607s
error Generating JavaScript bundles failed

Steps to reproduce

This is the repo

Expected result

It should build.

Actual result

It failed to build

Environment

System: OS: macOS 10.15.6 CPU: (8) x64 Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz Shell: 5.7.1 - /bin/zsh Binaries: Node: 14.1.0 - ~/.nvm/versions/node/v14.1.0/bin/node Yarn: 1.22.4 - ~/code/jmfservices/node_modules/.bin/yarn npm: 6.14.6 - ~/.nvm/versions/node/v14.1.0/bin/npm Languages: Python: 2.7.16 - /usr/bin/python Browsers: Chrome: 84.0.4147.89 Firefox: 78.0.2 Safari: 13.1.2 npmPackages: gatsby: ^2.20.25 => 2.20.25 gatsby-background-image: ^1.1.1 => 1.1.1 gatsby-image: ^2.3.4 => 2.3.4 gatsby-plugin-algolia: ^0.11.0 => 0.11.0 gatsby-plugin-codegen: ^1.2.0 => 1.2.0 gatsby-plugin-emotion: ^4.2.2 => 4.2.2 gatsby-plugin-feed: ^2.4.3 => 2.4.3 gatsby-plugin-firebase: ^0.2.0-beta.4 => 0.2.0-beta.4 gatsby-plugin-google-analytics: ^2.2.4 => 2.2.4 gatsby-plugin-manifest: ^2.3.5 => 2.3.5 gatsby-plugin-nprogress: ^2.3.4 => 2.3.4 gatsby-plugin-offline: ^3.1.4 => 3.1.4 gatsby-plugin-react-helmet: ^3.2.4 => 3.2.4 gatsby-plugin-sharp: ^2.5.6 => 2.5.6 gatsby-plugin-typescript: ^2.3.3 => 2.3.3 gatsby-plugin-typography: ^2.4.3 => 2.4.3 gatsby-remark-copy-linked-files: ^2.2.3 => 2.2.3 gatsby-remark-images: ^3.2.4 => 3.2.4 gatsby-remark-prismjs: ^3.4.3 => 3.4.3 gatsby-remark-responsive-iframe: ^2.3.3 => 2.3.3 gatsby-remark-smartypants: ^2.2.3 => 2.2.3 gatsby-source-airtable: ^2.1.0 => 2.1.0 gatsby-source-filesystem: ^2.2.4 => 2.2.4 gatsby-transformer-remark: ^2.8.23 => 2.8.23 gatsby-transformer-sharp: ^2.4.6 => 2.4.6

pvdz commented 4 years ago

I would suggest taking a look at the error that is reported in circle.

It may be necessary to drop the cache to reproduce this locally (gatsby clean).

From the repro, from circle:

> gatsby build

success open and validate gatsby-configs - 2.153s
success load plugins - 2.274s
success onPreInit - 0.008s
success delete html and css files from previous builds - 0.010s
success initialize cache - 0.007s
success copy gatsby files - 0.033s
success onPreBootstrap - 0.007s
success createSchemaCustomization - 0.004s
warning 
API key is required to connect to Airtable
warning The gatsby-source-airtable plugin has generated no Gatsby nodes. Do you need it?
success source and transform nodes - 0.125s
success building schema - 0.356s
error There was an error in your GraphQL query:

Expected type airtableManufacturerFieldsEnum, found data___Manufacturer. Did you mean the enum value Manufacturer?
error There was an error in your GraphQL query:

Cannot query field "recordId" on type "airtableManufacturer".

Cannot query field "recordId" on type "airtableManufacturer".

If you don't expect "recordId" to exist on the type "airtableManufacturer" it is most likely a typo.
However, if you expect "recordId" to exist there are a couple of solutions to common problems:

- If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
- The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
- You want to optionally use your field "recordId" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")

It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "airtableManufacturer":
https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions
error There was an error in your GraphQL query:

Cannot query field "data" on type "airtableManufacturer".

If you don't expect "data" to exist on the type "airtableManufacturer" it is most likely a typo.
However, if you expect "data" to exist there are a couple of solutions to common problems:

- If you added a new data source and/or changed something inside gatsby-node.js/gatsby-config.js, please try a restart of your development server
- The field might be accessible in another subfield, please try your query in GraphiQL and use the GraphiQL explorer to see which fields you can query and what shape they have
- You want to optionally use your field "data" and right now it is not used anywhere. Therefore Gatsby can't infer the type and add it to the GraphQL schema. A quick fix is to add a least one entry with that field ("dummy content")

It is recommended to explicitly type your GraphQL schema if you want to use optional fields. This way you don't have to add the mentioned "dummy content". Visit our docs to learn how you can define the schema for "airtableManufacturer":
https://www.gatsbyjs.org/docs/schema-customization/#creating-type-definitions
not finished createPages - 0.082s
rmcsharry commented 4 years ago

Thanks for taking a look but that is not the correct log. That is coming from the Github Action - which as you can see, cannot find the Airtable API key (even though it is set in a github secret).

So I had to abandon using Github Actions (you will see the repo no longer has them activated) and I moved instead to Circle CI.

As you can see below in the Circle CI log, it does correctly get the Airtable API key and fetches the data from Airtable.

(I have no idea why Github Actions could not find the key, and could find no info on it anywhere).

Here is the log from CircleCI:

> gatsby build

success open and validate gatsby-configs - 1.438s
success load plugins - 1.881s
success onPreInit - 0.007s
success delete html and css files from previous builds - 0.008s
success initialize cache - 0.007s
success copy gatsby files - 0.045s
success onPreBootstrap - 0.007s
success createSchemaCustomization - 0.006s

fetch all Airtable rows from 4 tables: 4995.873ms
success source and transform nodes - 6.757s
success building schema - 0.433s
success createPages - 0.716s
success createPagesStatefully - 0.089s
success onPreExtractQueries - 0.001s
success update schema - 0.051s
success extract queries from components - 0.517s
success write out requires - 0.015s
success write out redirect data - 0.001s
warning The icon(src/content/assets/BibliotechLogoIcon.jpg) you provided to 'gatsby-plugin-manifest' is not square.
The icons we generate will be square and for the best results we recommend you provide a square icon.

success Build manifest and related icons - 0.101s
success [gatsby-plugin-codegen] saved localSchemaFile: ./schema.json
success [gatsby-plugin-codegen] saved apollo config: apollo.config.js
  ✔ Loading Apollo Project
  ✔ Generating query files with 'typescript' target - wrote 35 files
success [gatsby-plugin-codegen] types for typescript generated
success onPostBootstrap - 0.686s
⠀
info bootstrap finished - 14.297 s
⠀
 [] 0/3 0.0 s 0% Generating image thumbnails [] 0/16 0.0 s 0% Generating image thumbnails [] 1/893 0.0 s 0% run queries [] 3/893 0.0 s 0% run queries [] 0/22 0.0 s 0% Generating image thumbnails [] 4/893 0.0 s 0% run queries [] 10/22 1.8 s 45% Generating image thumbnails [] 16/22 2.7 s 72% Generating image thumbnails [] 22/22 3.7 s 100% Generating image thumbnailsfailed Building production JavaScript and CSS bundles - 42.607s
error Generating JavaScript bundles failed

Can't resolve '../../../contexts/Firebase' in '/home/circleci/gatsby-site/src/components/Forms/ContactUs'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/Favorites' in '/home/circleci/gatsby-site/src/components/ManufacturerList'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/Firebase' in '/home/circleci/gatsby-site/src/components/FavoriteButton'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/Firebase' in '/home/circleci/gatsby-site/src/components/FavoritesHeader'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/Firebase' in '/home/circleci/gatsby-site/src/components/Header'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/Firebase' in '/home/circleci/gatsby-site/src/components/Home'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/Firebase' in '/home/circleci/gatsby-site/src/components/Layout'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/Firebase' in '/home/circleci/gatsby-site/src/components/LogOutButton'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/Firebase' in '/home/circleci/gatsby-site/src/components/SignInWithGoogle'
error Generating JavaScript bundles failed

Can't resolve '../contexts/Favorites' in '/home/circleci/gatsby-site/src/templates'
error Generating JavaScript bundles failed

Can't resolve '../contexts/Firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/Firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/Firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/Firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/Firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/Firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/Firebase' in '/home/circleci/gatsby-site/src/templates'
error Generating JavaScript bundles failed

Can't resolve './Input' in '/home/circleci/gatsby-site/src/components/SearchBox'
error Generating JavaScript bundles failed

Can't resolve './src/contexts/GlobalContextProvider' in '/home/circleci/gatsby-site'
not finished run queries - 44.556s
not finished Generating image thumbnails - 44.470s
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! Bibliotech@0.1.0 build: `gatsby build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the Bibliotech@0.1.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2020-07-27T15_00_37_467Z-debug.log

Exited with code exit status 1
CircleCI received exit code 1
pvdz commented 4 years ago

Ah, that's a casing issue. Your contexts folder is actually called Contexts on disk. You wouldn't see this locally if you're on a case insensitive OS like Windows, but linux absolutely cares and considers foo/bar and Foo/bar two different paths. (ht. @pieh)

I'm confident that if you change the reference to use upper cased Contexts that it will resolve this issue.

Also would suggest to either be very consistent in casing folder/file names, or to choose to always use lower case file names regardless. There ought to have been a warning in the terminal related to this when you do gatsby develop.

Going to close this, please re-open if that was not the issue :)

rmcsharry commented 4 years ago

@pvdz I changed all folders and filenames to be lowercase and unfortunately the same error of "Generating Image Thumbnails" appears.

Locally it builds the images fine (on MacOS):

success run queries - 39.847s - 892/892 22.39/s
[============================]   46.473 s 4/4 100% Generating image thumbnails
[=======================     ]   6.524 s 750/888 84% Building static HTML for pages
success Building static HTML for pages - 7.025s - 888/888 126.40/s
success Generating image thumbnails - 46.982s - 4/4 0.09/s

But CircleCI still fails.

I find the error report strange/confusing, because the log stays it got to 22/22 100% generating image thumbnails:

Generating image thumbnails [] 22/22 4.0 s 100%

but then immediately afterwards prints failed. Then at the very end of the log:

not finished Generating image thumbnails - 40.638s

Here is the full log:

#!/bin/bash -eo pipefail
npm run build

> Bibliotech@0.1.0 build /home/circleci/gatsby-site
> gatsby build

success open and validate gatsby-configs - 1.567s
success load plugins - 1.932s
success onPreInit - 0.007s
success delete html and css files from previous builds - 0.009s
success initialize cache - 0.192s
success copy gatsby files - 0.035s
success onPreBootstrap - 0.007s
success createSchemaCustomization - 0.004s

fetch all Airtable rows from 4 tables: 1892.549ms
success source and transform nodes - 3.789s
success building schema - 0.510s
success createPages - 0.796s
success createPagesStatefully - 0.100s
success onPreExtractQueries - 0.001s
success update schema - 0.051s
success extract queries from components - 0.808s
success write out requires - 0.011s
success write out redirect data - 0.001s
warning The icon(src/content/assets/bibliotech-logo-icon.jpg) you provided to 'gatsby-plugin-manifest' is not square.
The icons we generate will be square and for the best results we recommend you provide a square icon.

success Build manifest and related icons - 0.121s
success [gatsby-plugin-codegen] saved localSchemaFile: ./schema.json
success [gatsby-plugin-codegen] saved apollo config: apollo.config.js
  ✔ Loading Apollo Project
  ✔ Generating query files with 'typescript' target - wrote 35 files
success [gatsby-plugin-codegen] types for typescript generated
success onPostBootstrap - 0.773s
⠀
info bootstrap finished - 12.337 s
⠀
 [] 0/3 0.0 s 0% 
Generating image thumbnails [] 0/16 0.0 s 0% 
Generating image thumbnails [] 1/892 0.0 s 0% 
run queries [] 2/892 0.0 s 0% 
run queries [] 0/22 0.0 s 0% 
Generating image thumbnails [] 4/892 0.0 s 0% 
run queries [] 10/22 1.9 s 45% 
Generating image thumbnails [] 16/22 3.0 s 72% 
Generating image thumbnails [] 22/22 4.0 s 100% 
Generating image thumbnails 
failed 
Building production JavaScript and CSS bundles - 38.460s
error Generating JavaScript bundles failed

Can't resolve '../../../contexts/firebase' in '/home/circleci/gatsby-site/src/components/forms/contact-us'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/favorites' in '/home/circleci/gatsby-site/src/components/manufacturer-list'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/firebase' in '/home/circleci/gatsby-site/src/components/Header'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/firebase' in '/home/circleci/gatsby-site/src/components/Home'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/firebase' in '/home/circleci/gatsby-site/src/components/Layout'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/firebase' in '/home/circleci/gatsby-site/src/components/favorite-button'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/firebase' in '/home/circleci/gatsby-site/src/components/favorites-header'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/firebase' in '/home/circleci/gatsby-site/src/components/log-out-button'
error Generating JavaScript bundles failed

Can't resolve '../../contexts/firebase' in '/home/circleci/gatsby-site/src/components/sign-in-with-google'
error Generating JavaScript bundles failed

Can't resolve '../body-sections/website' in '/home/circleci/gatsby-site/src/components/large-card'
error Generating JavaScript bundles failed

Can't resolve '../contexts/favorites' in '/home/circleci/gatsby-site/src/templates'
error Generating JavaScript bundles failed

Can't resolve '../contexts/firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/firebase' in '/home/circleci/gatsby-site/src/pages'
error Generating JavaScript bundles failed

Can't resolve '../contexts/firebase' in '/home/circleci/gatsby-site/src/templates'
error Generating JavaScript bundles failed

Can't resolve './Input' in '/home/circleci/gatsby-site/src/components/search-box'
error Generating JavaScript bundles failed

Can't resolve './rep' in '/home/circleci/gatsby-site/src/components/body-sections/tech-reps'
error Generating JavaScript bundles failed

Can't resolve './restricted' in '/home/circleci/gatsby-site/src/components/small-card'
error Generating JavaScript bundles failed

Can't resolve './src/contexts/global-context-provider' in '/home/circleci/gatsby-site'
not finished run queries - 40.738s
not finished Generating image thumbnails - 40.638s
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! Bibliotech@0.1.0 build: `gatsby build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the Bibliotech@0.1.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2020-08-04T16_44_24_953Z-debug.log

Exited with code exit status 1
CircleCI received exit code 1
rmcsharry commented 4 years ago

@pvdz Please can you re-open this? Thanks.

rmcsharry commented 4 years ago

For anyone else landing here, the error "Generating image thumbnails failed" is mis-leading. The error immediately afterwards, where it says "Generating JavaScript bundles failed" is the actual problem.

Then it proceeds to list the things it cannot resolve. Locally they can be resolved (as it's the same OS, but on Circle CI, which is Linux, it cannot resolve the names - even though, as suggested by @pvdz, I renamed everything to lowercase).

I destroyed the .cache and .public folders for all previous builds, still no joy.

So then I went to each file or folder that it listed in the log and gave it a completely new name.

For example: Can't resolve './restricted' in '/home/circleci/gatsby-site/src/components/small-card'

I renamed './restricted' to './unauthorised'. That error went away.

After renaming all the files it could not resolve, the problem was fixed.

So, somehow, Gatsby is storing the filenames somewhere and NOT properly updating them, despite renaming and despite deleting .cache and .public. How is that possible? I'm stumped.

Proof, here is the new log - only 1 rename to go! WTH?!!!!!!!!!!!


#!/bin/bash -eo pipefail
npm run build

> Bibliotech@0.1.0 build /home/circleci/gatsby-site
> gatsby build

success open and validate gatsby-configs - 1.669s
success load plugins - 2.009s
success onPreInit - 0.001s
success delete html and css files from previous builds - 0.008s
success initialize cache - 0.004s
success copy gatsby files - 0.030s
success onPreBootstrap - 0.013s
success createSchemaCustomization - 0.006s

fetch all Airtable rows from 4 tables: 1457.475ms
success source and transform nodes - 3.298s
success building schema - 0.439s
success createPages - 0.719s
success createPagesStatefully - 0.095s
success onPreExtractQueries - 0.000s
success update schema - 0.065s
success extract queries from components - 0.499s
success write out requires - 0.007s
success write out redirect data - 0.000s
warning The icon(src/content/assets/bibliotech-logo-icon.jpg) you provided to 'gatsby-plugin-manifest' is not square.
The icons we generate will be square and for the best results we recommend you provide a square icon.

success Build manifest and related icons - 0.107s
success [gatsby-plugin-codegen] saved localSchemaFile: ./schema.json
success [gatsby-plugin-codegen] saved apollo config: apollo.config.js
  ✔ Loading Apollo Project
  ✔ Generating query files with 'typescript' target - wrote 35 files
success [gatsby-plugin-codegen] types for typescript generated
success onPostBootstrap - 0.846s
⠀
info bootstrap finished - 11.518 s
⠀
 [] 0/3 0.0 s 0% Generating image thumbnails [] 0/10 0.0 s 0% Generating image thumbnails [] 1/896 0.0 s 0% run queries [] 2/896 0.0 s 0% run queries [] 3/896 0.0 s 0% run queries [] 0/13 0.0 s 0% Generating image thumbnails [] 4/13 0.3 s 30% Generating image thumbnails [] 4/896 0.0 s 0% run queries [] 7/13 1.6 s 53% Generating image thumbnails [] 10/13 3.1 s 76% Generating image thumbnails [] 13/13 4.1 s 100% Generating image thumbnailsfailed Building production JavaScript and CSS bundles - 43.695s
error Generating JavaScript bundles failed

Can't resolve './src/contexts/global-context-provider' in '/home/circleci/gatsby-site'
not finished run queries - 45.833s
not finished Generating image thumbnails - 45.714s
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! Bibliotech@0.1.0 build: `gatsby build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the Bibliotech@0.1.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2020-08-31T22_09_01_063Z-debug.log

Exited with code exit status 1
CircleCI received exit code 1