Issue: warning Query takes too long + Build breaks on netlify

[quansenB]

Description

I'm getting the Warning Query takes too long on very small queries, for example:

export const pageQuery = graphql`
  query impressumPageQuery {
    markdownRemark(frontmatter: { templateKey: { eq: "impressum-page" } }) {
      html
    }
  }
`;

This is not a problem for building my site locally but is on netlify which enforces a RAM limit (NPM error code 137).

I have no idea why such a query would be so big. At first I thought it might be related to the new gatsby-plugin-image but it also appears on some sites with very small queries, like the above.

Steps to reproduce

1. Fork project: https://github.com/quansenB/Website-Faltermeier
2. Build on netlify

Expected result

Should be built without Error and query takes too long warning

Actual result

Getting npm ERR! code 137 (out of memory)

Environment

System:
    OS: macOS 10.15.7
    CPU: (4) x64 Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 14.15.4 - /usr/local/bin/node
    npm: 6.14.10 - /usr/local/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 89.0.4389.82
    Edge: 89.0.774.54
    Firefox: 85.0.2
    Safari: 14.0.3
  npmPackages:
    gatsby: ^3.0.4 => 3.0.4
    gatsby-cli: ^3.0.0 => 3.0.0
    gatsby-plugin-google-tagmanager: ^3.0.0 => 3.0.0
    gatsby-plugin-image: ^1.0.1 => 1.0.1
    gatsby-plugin-manifest: ^3.0.0 => 3.0.0
    gatsby-plugin-minify-html: ^1.0.4 => 1.0.4
    gatsby-plugin-netlify: ^3.0.0 => 3.0.0
    gatsby-plugin-netlify-cms: ^5.0.0 => 5.0.0
    gatsby-plugin-offline: ^4.0.0 => 4.0.0
    gatsby-plugin-postcss: ^4.0.0 => 4.0.0
    gatsby-plugin-preact: ^5.0.0 => 5.0.0
    gatsby-plugin-preload-fonts: ^2.0.0 => 2.0.0
    gatsby-plugin-react-helmet: ^4.0.0 => 4.0.0
    gatsby-plugin-remove-serviceworker: ^1.0.0 => 1.0.0
    gatsby-plugin-robots-txt: ^1.5.5 => 1.5.5
    gatsby-plugin-sharp: ^3.0.1 => 3.0.1
    gatsby-plugin-sitemap: ^3.0.0 => 3.0.0
    gatsby-plugin-svgr: ^3.0.0-beta.0 => 3.0.0-beta.0
    gatsby-remark-images: ^4.0.0 => 4.0.0
    gatsby-remark-relative-images-v2: ^0.1.5 => 0.1.5
    gatsby-source-filesystem: ^3.0.0 => 3.0.0
    gatsby-transformer-remark: ^3.0.0 => 3.0.0
    gatsby-transformer-sharp: ^3.0.0 => 3.0.0
  npmGlobalPackages:
    gatsby: 3.0.4

[mrkaluzny]

I have the same error. The other problem is that gatsby-plugin-image seems to be generating the images when query is running. My builds are now 25min+ instead of previous 5-6min. Any help?

"dependencies": {
    "babel-preset-gatsby": "^1.0.0",
    "classnames": "^2.2.6",
    "gatsby": "^3.0.4",
    "gatsby-plugin-advanced-sitemap": "^1.4.5",
    "gatsby-plugin-brotli": "^2.0.0",
    "gatsby-plugin-disqus": "^1.2.2",
    "gatsby-plugin-gdpr-cookies": "^1.0.3",
    "gatsby-plugin-google-tagmanager": "^3.0.0",
    "gatsby-plugin-image": "^1.0.1",
    "gatsby-plugin-netlify": "^3.0.0",
    "gatsby-plugin-netlify-cms": "^5.0.0",
    "gatsby-plugin-purgecss": "^6.0.0",
    "gatsby-plugin-react-helmet": "^4.0.0",
    "gatsby-plugin-root-import": "^2.0.5",
    "gatsby-plugin-sass": "^4.0.2",
    "gatsby-plugin-sharp": "^3.0.1",
    "gatsby-remark-autolink-headers": "^3.0.0",
    "gatsby-remark-copy-linked-files": "^3.0.0",
    "gatsby-remark-images": "^4.0.0",
    "gatsby-remark-prismjs": "^4.0.0",
    "gatsby-remark-relative-images": "^2.0.2",
    "gatsby-source-filesystem": "^3.0.0",
    "gatsby-transformer-remark": "^3.0.0",
    "gatsby-transformer-sharp": "^3.0.0",
    "lodash": "^4.17.15",
    "lodash-webpack-plugin": "^0.11.4",
    "netlify-cms-app": "^2.14.29",
    "node-sass": "^5.0.0",
    "parcel-bundler": "^1.9.4",
    "prismjs": "^1.22.0",
    "prop-types": "^15.6.0",
    "react": "^17.0.1",
    "react-confetti": "^6.0.0",
    "react-dom": "^17.0.1",
    "react-helmet": "^6.1.0",
    "react-markdown": "^5.0.2",
    "react-slick": "^0.28.1",
    "react-svg": "^12.1.0",
    "react-transition-group": "^4.4.1",
    "slick-carousel": "^1.8.1",
    "universal-cookie": "^4.0.2",
    "uuid": "^8.3.2"
  },

[bradgarropy]

Same here. I updated to Gatsby v3 and my build went from 3 minutes to 30 minutes with a timeout on Netlify. I see many of these messages.

10:36:39 AM: warning Query takes too long:
10:36:39 AM: File path: /opt/build/repo/src/templates/post.js
10:36:39 AM: URL path: /posts/to-be-loved-by-you
10:36:39 AM: Context: {
10:36:39 AM:     "slug": "to-be-loved-by-you"
10:36:39 AM: }

The website is open source. In particular, the build logs are complaining about the page query in my post.js template.

const query = graphql`
    query($slug: String!) {
        markdownRemark(frontmatter: {slug: {eq: $slug}}) {
            html
            frontmatter {
                slug
                title
                date(formatString: "MMMM D, YYYY")
                tags
                image {
                    publicURL
                    childImageSharp {
                        gatsbyImageData(
                            width: 750
                            placeholder: BLURRED
                            formats: [AUTO, WEBP, AVIF]
                            quality: 100
                        )
                    }
                }
            }
        }
    }
`

[creotip]

i have the same problem.

[quansenB]

This was assigned topic: scaling builds, so just for information: This is a rather small site. Like 15 pages total, and the queries in question query just some markdown and one image max.

[vladar]

We've heard reports that AVIF transformations are slow. Can you double-check by temporarily disabling any AVIF usages and build without it? Just to confirm if it is the actual source of the issue.

[lomby92]

I will try it tomorrow, thanks for the suggestion

[mrkaluzny]

@vladar That's definitely the case, another point could be the DOMINANT_COLOR option. As a solution I had to switch from Netlify to Gatsby Cloud for build processes (Netlify would finish builds but threw errors when deploying).

When it comes to images, could you also disable warnings on FULL_WIDTH images? It seems kind of redundant ;)

[bradgarropy]

@vladar I took your advice and simplified my gatsbyImageData query from this:

childImageSharp {
    gatsbyImageData(
        width: 750
        placeholder: BLURRED
        formats: [AUTO, WEBP, AVIF]
        quality: 100
    )
}

Down to this.

childImageSharp {
    gatsbyImageData(width: 750)
}

But, I saw no changes or improvements in the build time. All of the query takes too long messages were still present.

9:21:47 PM: warning Query takes too long:
9:21:47 PM: File path: /opt/build/repo/src/templates/post.js
9:21:47 PM: URL path: /posts/bury-my-bones-heart-wrenching-music-video
9:21:47 PM: Context: {
9:21:47 PM:     "slug": "bury-my-bones-heart-wrenching-music-video"
9:21:47 PM: }
9:21:47 PM: warning Query takes too long:
9:21:47 PM: File path: /opt/build/repo/src/templates/post.js
9:21:47 PM: URL path: /posts/casey-donahews-nye-livestream
9:21:47 PM: Context: {
9:21:47 PM:     "slug": "casey-donahews-nye-livestream"
9:21:47 PM: }
9:21:47 PM: warning Query takes too long:
9:21:47 PM: File path: /opt/build/repo/src/templates/post.js
9:21:47 PM: URL path: /posts/evan-felker-finds-sobriety
9:21:47 PM: Context: {
9:21:47 PM:     "slug": "evan-felker-finds-sobriety"
9:21:47 PM: }
9:21:47 PM: warning Query takes too long:
9:21:47 PM: File path: /opt/build/repo/src/templates/post.js
9:21:47 PM: URL path: /posts/country-bars-closing
9:21:47 PM: Context: {
9:21:47 PM:     "slug": "country-bars-closing"
9:21:47 PM: }

[lomby92]

I haven't done the test without AVIF yet.

As I have both Gatsby cloud build and Netlify build on the same site, I can report the differences:

Gatsby Cloud: success run page queries - 0.347s - 7/7 20.19/s Netlify: success run page queries - 167.913s - 7/7 0.04/s

Note that on Gatsby Cloud the Query takes too long warning does not happen

[vladar]

@lomby92 Image processing in Gatsby Cloud is heavily optimized. So this could explain the difference if the source of the issue is AVIF in your case. Curious to see your results without AVIF to confirm.

@bradgarropy This happens locally for you too? If I run it locally - will I see the same issue? Or does it only happen on Netlify?

[vladar]

We may be seeing this sharp issue in play here: https://github.com/lovell/sharp/issues/2597 (not 100% sure yet)

There is a work in progress to improve it though.

[lomby92]

@vladar thanks for the info about the Gatsby Cloud optimization.

After a local test I was able to reproduce the warning disabling the AVIF format.

Without AVIF

success run page queries - 1.361s - 8/8 5.88/s
success write out requires - 0.003s
success Building production JavaScript and CSS bundles - 31.376s
success Rewriting compilation hashes - 0.004s
success Building HTML renderer - 3.590s
success Building static HTML for pages - 0.150s - 8/8 53.50/s
success onPostBuild - 0.016s
info Done building in 42.339797475 sec

With AVIF

success run page queries - 31.345s - 8/8 0.26/s
success write out requires - 0.011s
success Building production JavaScript and CSS bundles - 34.760s
success Rewriting compilation hashes - 0.003s
success Building HTML renderer - 4.159s
success Building static HTML for pages - 0.214s - 8/8 37.44/s
success onPostBuild - 0.020s
info Done building in 81.095961358 sec

Transforming images into AVIF format uses all my CPU 6c/12t at 100%. Also, in this case I can see the warnings as expected.

Conclusions

As you can see, the AVIF setting lead to a longer build, it's confirmed

[quansenB]

I can confirm that the build works without AVIF. Though I'm still getting the Query takes to long warning.

I'm now also getting an seemingly unrelated error, which also doesn't appear locally. (The build passes though). I'm dropping it here, just in case it helps:

error (node:1307) [DEP_WEBPACK_COMPILATION_NORMAL_MODULE_LOADER_HOOK] DeprecationWarning: Compilation.hooks.normalModuleLoader was moved to NormalModule.getCompilationHooks(compilation).loader
1:58:38 PM: (Use `node --trace-deprecation ...` to show where the warning was created)
1:58:55 PM: error (node:1307) [DEP_WEBPACK_COMPILATION_ASSETS] DeprecationWarning: Compilation.assets will be frozen in future, all modifications are deprecated.
1:58:55 PM: BREAKING CHANGE: No more changes should happen to Compilation.assets after sealing the Compilation.
1:58:55 PM:     Do changes to assets earlier, e. g. in Compilation.hooks.processAssets.
1:58:55 PM:     Make sure to select an appropriate stage from Compilation.PROCESS_ASSETS_STAGE_*.

[bradgarropy]

@vladar This only happens on Netlify, locally things run just fine!

[ascorbic]

I'm going to add a note to the docs warning about the build time performance implications of AVIF support. It's the reason we don't enable it by default. I think the issue here is that Netlify containers don't appear to have enough memory to handle AVIF builds on your site, whereas Gatbsy Cloud is specifically optimised to handle it.

[bradgarropy]

That sounds great @ascorbic! Alghtouh I'm still seeing the query takes too long issue even without AVIFs included.

[ehowey]

Just hopping in to confirm that I am seeing this error as well, build is still completing in my case but I am testing it on just my personal website. As mentioned above I was also seeing the deprecation warnings about Webpack [DEP_WEBPACK_COMPILATION_NORMAL_MODULE_LOADER_HOOK]. I can confirm I am not using AVIF.

[vladar]

@ehowey can you share a reproduction project for Query takes too long without AVIF? I'll try to figure out what's going on.

[ehowey]

@vladar It seems to have gone away now - at least in the most recent builds on Gatsby v3.2. I'll pop back in here if I see it again.

Since that time I have done some optimizations to my GraphQL queries for the images so I may have even solved it myself. One thing I had happening early in my transition to Gatsby v3 and gatsby-plugin-image was queries that were just gatsbyImageData without any options passed to it, so maybe that was a problem?

[ascorbic]

@ehowey Were the source images very large? That's the only scenario where I could see having it without options could make it slow, as it would default to the full size with constrained layout.

[bradgarropy]

@vladar After updating to v3.2.0 I'm still experiencing the same Query takes too long messages. https://app.netlify.com/sites/dailytexascountry/deploys/6065198d1239431a00670966

[ehowey]

@ascorbic could be - sometimes I take images straight from Unsplash and don't bother downsizing them because Gatsby has usually handled that for me.

[ardiewen]

Just wanted to comment here that I'm experiencing this with local builds on both a 2017 macbook pro (4 core / 16gb ram) and a remote Ubuntu development server with 4 cores / 8gb ram. No issues with Gatsby Cloud, and we don't build on Netlify anymore (but we do deploy there for Netlify functions).

We have image fragments with formats: [AUTO, WEBP, AVIF] and we have quite a few images (400-500) that are in high quality.

When I gatsby clean && gatsby develop there's no issues with building the development bundle and proceeding as intended. If however, I don't run gatsby clean it will usually hang on the run page queries step.

When I gatsby clean && gatsby build, I will usually hang on one of the following:

MBP:

success run page queries - 6.833s - 1/1 0.15/s

Ubuntu:

success Building production JavaScript and CSS bundles - 40.024s

success Rewriting compilation hashes - 0.001s

MBP Info:

  System:
    OS: macOS 11.2.3
    CPU: (4) x64 Intel(R) Core(TM) i5-7267U CPU @ 3.10GHz
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.15.3 - ~/.nvm/versions/node/v14.15.3/bin/node
    Yarn: 1.22.10 - ~/Projects/WORKING_DIRECTORY/node_modules/.bin/yarn
    npm: 6.14.9 - ~/.nvm/versions/node/v14.15.3/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 89.0.4389.128
    Firefox: 86.0
    Safari: 14.0.3
  npmPackages:
    gatsby: ^3.3.0 => 3.3.0
    gatsby-omni-font-loader: ^1.3.0 => 1.3.1
    gatsby-plugin-image: ^1.3.0 => 1.3.0
    gatsby-plugin-mailchimp: ^5.2.2 => 5.2.2
    gatsby-plugin-manifest: ^3.2.0 => 3.3.0
    gatsby-plugin-netlify: ^3.2.0 => 3.3.0
    gatsby-plugin-no-sourcemaps: ^2.2.0 => 2.9.0
    gatsby-plugin-react-helmet: ^4.2.0 => 4.3.0
    gatsby-plugin-react-svg: ^2.1.2 => 2.1.2
    gatsby-plugin-remove-serviceworker: ^1.0.0 => 1.0.0
    gatsby-plugin-segment-js: ^3.6.0 => 3.6.0
    gatsby-plugin-sharp: ^3.3.0 => 3.3.0
    gatsby-plugin-sitemap: ^3.2.0 => 3.3.0
    gatsby-plugin-styled-components: ^4.2.0 => 4.3.0
    gatsby-plugin-webpack-bundle-analyser-v2: ^1.1.16 => 1.1.22
    gatsby-source-filesystem: ^3.2.0 => 3.3.0
    gatsby-source-wordpress: ^5.3.0 => 5.3.0
    gatsby-transformer-sharp: ^3.3.0 => 3.3.0

Ubuntu VM Info

  System:
    OS: Linux 4.15 Ubuntu 18.04.2 LTS (Bionic Beaver)
    CPU: (4) x64 Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
    Shell: 5.4.2 - /usr/bin/zsh
  Binaries:
    Node: 14.16.1 - ~/.nvm/versions/node/v14.16.1/bin/node
    Yarn: 1.22.10 - ~/Projects/WORKING_DIRECTORY/node_modules/.bin/yarn
    npm: 6.14.12 - ~/.nvm/versions/node/v14.16.1/bin/npm
  Languages:
    Python: 2.7.17 - /usr/bin/python
  npmPackages:
    gatsby: ^3.3.0 => 3.3.0
    gatsby-omni-font-loader: ^1.3.0 => 1.3.1
    gatsby-plugin-image: ^1.3.0 => 1.3.0
    gatsby-plugin-mailchimp: ^5.2.2 => 5.2.2
    gatsby-plugin-manifest: ^3.2.0 => 3.3.0
    gatsby-plugin-netlify: ^3.2.0 => 3.3.0
    gatsby-plugin-no-sourcemaps: ^2.2.0 => 2.9.0
    gatsby-plugin-react-helmet: ^4.2.0 => 4.3.0
    gatsby-plugin-react-svg: ^2.1.2 => 2.1.2
    gatsby-plugin-remove-serviceworker: ^1.0.0 => 1.0.0
    gatsby-plugin-segment-js: ^3.6.0 => 3.6.0
    gatsby-plugin-sharp: ^3.3.0 => 3.3.0
    gatsby-plugin-sitemap: ^3.2.0 => 3.3.0
    gatsby-plugin-styled-components: ^4.2.0 => 4.3.0
    gatsby-plugin-webpack-bundle-analyser-v2: ^1.1.16 => 1.1.22
    gatsby-source-filesystem: ^3.2.0 => 3.3.0
    gatsby-source-wordpress: ^5.3.0 => 5.3.0
    gatsby-transformer-sharp: ^3.3.0 => 3.3.0

[pfortes]

I had some posts with the "Query takes too long" messages.

it seams like having HTML tags in my mdx files seamed to be the culprit of my long querys, i dont know if the classes or the dimension atributes where afecting.

Changing images from something like <img className="alignnone size-medium wp-image-574" alt="logo" src="proyeccionLentes3d-logo-300x201.png" width="250" />

To: ![logo](proyeccionLentes3d-logo-300x201.png)

Stoped the messages from showing.

My build changed from:

• 32min 55sec with avif activated
• 8min 20sec without avif but with the long query messages
• 7min 3sec without the img tags in the mdx

I think using avif rirght now is not a good idea for me!

PD: i only have 63 pages..

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[jodiedoubleday]

I'm getting this on all my recent builds on Netlify, its take my build times from ~5 minutes to 30+ minutes. I'm using gatsby-image as I can't upgrade yet.

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions[bot]]

Hey again!

It’s been 60 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it. Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to comment on this issue or create a new one if you need anything else. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜