improve error message "gatsby-source-wordpress Request failed with status code 500"

[amcc]

I have started getting the error gatsby-source-wordpress Request failed with status code 500

This has been intermittent, and everything was working fine until recently. I assume it might be because the number of posts has grown beyond a certain amount, but its relatively small - around 50 posts and users. The full error is below:

success  gatsby-source-wordpress  writing GraphQL queries to disk at ./WordPress/GraphQL/ - 0.157s
success  gatsby-source-wordpress  ingest WPGraphQL schema - 2.064s
success createSchemaCustomization - 2.736s
success  gatsby-source-wordpress  Comment - 0.633s - fetched 0
success  gatsby-source-wordpress  Page - 0.631s - fetched 2
success  gatsby-source-wordpress  ContentType - 0.856s - fetched 3
success  gatsby-source-wordpress  Taxonomy - 0.863s - fetched 3
success  gatsby-source-wordpress  Menu - 0.924s - fetched 0
success  gatsby-source-wordpress  MenuItem - 0.960s - fetched 0
success  gatsby-source-wordpress  Tag - 0.937s - fetched 13
success  gatsby-source-wordpress  Category - 1.366s - fetched 15
success  gatsby-source-wordpress  Post - 1.378s - fetched 10

 ERROR 

 gatsby-source-wordpress  Request failed with status code 500

Error occured while fetching nodes of the "User" type.

GraphQL request to https://myserver.com/graphql failed.

Please ensure the following statements are true
  - your WordPress URL is correct in gatsby-config.js
  - your server is responding to requests
  - WPGraphQL and WPGatsby are installed in your WordPress backend

not finished source and transform nodes - 3.992s
not finished  gatsby-source-wordpress  fetching nodes - 3.944s - 46 total
not finished  gatsby-source-wordpress  User - 2.513s
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! using-gatsby-source-wordpress-experimental@0.0.1 clean-develop: `gatsby clean && gatsby develop`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the using-gatsby-source-wordpress-experimental@0.0.1 clean-develop script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/me/.npm/_logs/2020-07-23T16_45_26_930Z-debug.log
The terminal process "/bin/bash '-c', 'npm run clean-develop'" terminated with exit code: 1.

I have tried using the following option in gatsby-config.js:

{
      resolve: `gatsby-source-wordpress-experimental`,
      options: {
        url: process.env.WPGRAPHQL_URL,
        concurrentRequests: 1,
    }
}

This seemed to make it work again at first - now it isn't again.

[TylerBarnes]

Hi @amcc , it's hard to tell what it could be from the description. Are you able to check your PHP/WP error logs and send those over? (tyler@gatsbyjs.com). Also if you're able to provide a reproduction repo that would be super helpful!

[amcc]

So i've 'fixed' it for now by excluding type User - i think I've ruled out server config problems after speaking to the support people - I'm paying for a good server with good support.

I have found PHP errors -

[22-Jul-2020 23:03:13 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /home/admingm1/public_html/wp-content/plugins/wp-graphql-develop/src/Model/Post.php on line 633
[22-Jul-2020 23:03:37 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /home/admingm1/public_html/wp-content/plugins/wp-graphql-develop/src/Model/Model.php on line 310
[23-Jul-2020 16:26:29 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /home/admingm1/public_html/wp-content/plugins/wp-graphql-develop/src/Model/Model.php on line 312
[22-Jul-2020 23:07:15 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /home/admingm1/public_html/wp-content/plugins/wp-graphql-develop/src/Model/Post.php on line 595
[23-Jul-2020 00:14:12 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /home/admingm1/public_html/wp-content/plugins/wp-graphql-develop/src/Model/Post.php on line 621

The above errors occurred at various times - I've not dug into the PHP yet to see what is on those lines - I've not managed to tie in the times with the problem occurring yet, but will. Its strange that the User bit was causing the issues - there's around 50 users, but perhaps one of them has done something strange.

I'm trying to test with more memory now - though am having trouble with the server config there - I'll let you know if this solves the issue.

[amcc]

I've upped the memory limit with wordpress in the wp-config.php file by adding this line:

define('WP_MEMORY_LIMIT', ini_get('512MB'));

That seems to have fixed the issue.

I'm using a 5 year old WP site, which perhaps has a lower memory limit than more recent installs (though its fully updated to the latest version. I hope this helps someone else.

I bet its hard to do, but I wonder if the actual PHP error can be passed through to Gatsby, the 500 error I suppose is often what happens when PHP freaks out.

[TylerBarnes]

Thanks for the update here @amcc ! Yeah that's a tough one for sure. As far as I'm aware we wont be able to pass the error through 🤔 but maybe the error message for 500 errors could explain some common scenarios and potential fixes.

@jasonbahl what do you think?

[TylerBarnes]

@amcc the one other thing you can do is lower the option options.schema.perPage. It's set to 100 by default (100 nodes fetched per request), but for servers with low memory this might be too many nodes in some cases. If you run into the issue again after having increased server memory, try lowering perPage to 80 or even 50.

[TylerBarnes]

By the way, the current plan is to write an algorithm for our node fetch script that automatically recovers and attempts to reconfigure itself for these cases https://github.com/gatsbyjs/gatsby-source-wordpress-experimental/issues/66

[amcc]

thanks for the help - thats really useful. I couldn't see documentation about these options - i think they're there for the original Wordpress plugin - but i see you've added the documentation tag to this, so that may help others. Great plugin though - thanks for the good work

[niklaspesch]

I keep getting this error over and over again, sometimes i just have to close vscode and open again and sometimes i have to switch from the vs code terminal to my gitbash and it work again. But uff.. thats not developer friendly. As soon as i change some custom fields the whole process starts from the beginning.

And yes i tried different things like the .env file with:

GATSBY_CONCURRENT_DOWNLOAD=25
GATSBY_CONCURRENT_DOWNLOAD=10
GATSBY_CONCURRENT_DOWNLOAD=5
GATSBY_CONCURRENT_DOWNLOAD=2

or schema per Page with different values.

Maybe i'm doing something wrong but I don't know what to do next.

npx gatsby develop
success open and validate gatsby-configs - 0.047s
success load plugins - 1.006s
success onPreInit - 0.031s
success initialize cache - 0.017s
success copy gatsby files - 0.188s
success onPreBootstrap - 0.015s
success  gatsby-source-wordpress  ensuring plugin requiremenst are met - 3.459s
⠀
info  gatsby-source-wordpress 

        This is either your first build or the cache was cleared.
        Please wait while your WordPress data is synced to your Gatsby cache.

        Maybe now's a good time to get up and stretch? :D

success  gatsby-source-wordpress  ingest WPGraphQL schema - 5.133s
success createSchemaCustomization - 8.635s
info  gatsby-source-wordpress  GraphQL vars: {"first":50,"url":"https://.../graphql","after":null}
error  gatsby-source-wordpress  Request failed with status code 503

Error occured while fetching nodes of the "SingleIndustryNews" type.

GraphQL request to https://.../graphql failed.

Please ensure the following statements are true 
  - your WordPress URL is correct in gatsby-config.js
  - your server is responding to requests
  - WPGraphQL and WPGatsby are installed in your WordPress backend
not finished source and transform nodes - 7.778s
not finished  gatsby-source-wordpress  fetch root fields - 7.797s
not finished  gatsby-source-wordpress  fetching nodes - 7.807s - 0 total
not finished  gatsby-source-wordpress  AcademicJournal - 7.852s
not finished  gatsby-source-wordpress  Category - 7.867s
not finished  gatsby-source-wordpress  Comment - 7.879s
not finished  gatsby-source-wordpress  CommitteeMember - 7.884s
not finished  gatsby-source-wordpress  ContentType - 7.893s
not finished  gatsby-source-wordpress  EventPage - 7.899s
not finished  gatsby-source-wordpress  File - 7.910s
not finished  gatsby-source-wordpress  SingleFoundationNews - 7.916s
not finished  gatsby-source-wordpress  Homepage - 7.924s
not finished  gatsby-source-wordpress  SingleIndustryNews - 7.930s
not finished  gatsby-source-wordpress  MenuItem - 7.940s
not finished  gatsby-source-wordpress  Menu - 7.946s
not finished  gatsby-source-wordpress  Page - 7.957s
not finished  gatsby-source-wordpress  PostFormat - 7.972s
not finished  gatsby-source-wordpress  Post - 7.978s
not finished  gatsby-source-wordpress  Speaker - 7.986s
not finished  gatsby-source-wordpress  Sponsor - 7.992s
not finished  gatsby-source-wordpress  Tag - 8.004s
not finished  gatsby-source-wordpress  Taxonomy - 8.018s
not finished  gatsby-source-wordpress  UserRole - 8.023s
not finished  gatsby-source-wordpress  User - 8.033s

gatsby-config.js

require("dotenv").config({
  path: `.env.${process.env.NODE_ENV}`,
})

module.exports = {
  siteMetadata: {
    title: `tinyML Foundation`,
    description: `Kick off your next, great Gatsby project with this default starter. This barebones starter ships with the main Gatsby configuration files you might need.`,
    author: `@gatsbyjs`,
  },
  plugins: [
    `gatsby-plugin-react-helmet`,
    `gatsby-plugin-sass`,
    {
      resolve: `gatsby-source-wordpress-experimental`,
      options: {
        schema: {
          perPage: 50,
        },
        develop: {
          nodeUpdateInterval: 3000,
          hardCacheMediaFiles: false,
        },
        production: {
          hardCacheMediaFiles: false,
        },
        debug: {
          // these settings are all the defaults,
          // remove them if you'd like
          graphql: {
            showQueryOnError: false,
            showQueryVarsOnError: true,
            copyQueryOnError: true,
            panicOnError: true,
            // a critical error is a WPGraphQL query that returns an error and no response data. Currently WPGQL will error if we try to access private posts so if this is false it returns a lot of irrelevant errors.
            onlyReportCriticalErrors: true,
          },
        },
        url: `https://.../graphql`,
        verbose: true,
      },
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `images`,
        path: `${__dirname}/src/images`,
      },
    },
    `gatsby-transformer-sharp`,
    `gatsby-plugin-sharp`,
    {
      resolve: `gatsby-plugin-manifest`,
      options: {
        name: `gatsby-starter-default`,
        short_name: `starter`,
        start_url: `/`,
        background_color: `#663399`,
        theme_color: `#663399`,
        display: `minimal-ui`,
        icon: `src/images/gatsby-icon.png`, // This path is relative to the root of the site.
      },
    },
    {
      resolve: `gatsby-plugin-google-fonts-v2`,
      options: {
        fonts: [
          {
            family: "Raleway",
            variable: true,
            weights: ["300..700"],
          },
        ],
      },
    },
    // this (optional) plugin enables Progressive Web App + Offline functionality
    // To learn more, visit: https://gatsby.dev/offline
    // `gatsby-plugin-offline`,
  ],
}

package.json

{
  "name": "gatsby-starter-default",
  "private": true,
  "description": "A simple starter to get up and developing quickly with Gatsby",
  "version": "0.1.0",
  "author": "Kyle Mathews <mathews.kyle@gmail.com>",
  "dependencies": {
    "cache-manager-fs-hash": "0.0.9",
    "foundation-sites": "^6.6.3",
    "framer-motion": "^2.6.13",
    "gatsby": "^2.27.5",
    "gatsby-background-image": "^1.1.1",
    "gatsby-cli": "^2.12.65",
    "gatsby-image": "^2.4.9",
    "gatsby-plugin-google-fonts": "^1.0.1",
    "gatsby-plugin-google-fonts-v2": "0.0.6",
    "gatsby-plugin-manifest": "^2.4.14",
    "gatsby-plugin-offline": "^3.2.13",
    "gatsby-plugin-prefetch-google-fonts": "^1.4.3",
    "gatsby-plugin-react-helmet": "^3.3.6",
    "gatsby-plugin-sass": "^2.3.12",
    "gatsby-plugin-sharp": "^2.6.14",
    "gatsby-plugin-typography": "^2.5.10",
    "gatsby-source-filesystem": "^2.3.14",
    "gatsby-source-graphql": "^2.6.2",
    "gatsby-source-tribe-events": "^1.0.1",
    "gatsby-source-wordpress": "^3.3.34",
    "gatsby-source-wordpress-experimental": "^3.3.1",
    "gatsby-source-wpgraphql-images": "^1.0.4",
    "gatsby-transformer-sharp": "^2.5.7",
    "lottie-web": "^5.7.3",
    "node-gyp": "^7.1.2",
    "node-sass": "^4.14.1",
    "prop-types": "^15.7.2",
    "react": "^16.12.0",
    "react-dom": "^16.12.0",
    "react-foundation": "^0.9.6",
    "react-helmet": "^6.1.0",
    "react-icons": "^3.10.0",
    "react-typography": "^0.16.19",
    "smooth-scroll": "^16.1.3",
    "styled-components": "^5.1.1",
    "theme-ui": "^0.3.1",
    "typeface-raleway": "0.0.75",
    "typography": "^0.16.19"
  },
  "devDependencies": {
    "prettier": "2.0.5"
  },
  "keywords": [
    "gatsby"
  ],
  "license": "0BSD",
  "scripts": {
    "build": "gatsby build",
    "develop": "gatsby develop",
    "format": "prettier --write \"**/*.{js,jsx,ts,tsx,json,md}\"",
    "start": "npm run develop",
    "serve": "gatsby serve",
    "clean": "gatsby clean",
    "test": "echo \"Write tests! -> https://gatsby.dev/unit-testing\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/gatsbyjs/gatsby-starter-default"
  },
  "bugs": {
    "url": "https://github.com/gatsbyjs/gatsby/issues"
  }
}

[acao]

@peshyyy in a situation like this, because requests to your configured graphql URL aren't even being received, the error message you see about checking your configured URL and whether the site is available on your network is the best advice we can give.

GATSBY_CONCURRENT_DOWNLOAD will not help you here, because no schema is even available to you yet. that's why you don't see that in this error message.

im guessing https://.../graphql is an alias? youll need to make sure this URL works and presents a graphql schema first. if /graphql is not present, but the base url is a valid wordpress URL, you may have an issue on the wordpress plugin side that the gatsby source plugin cannot detect

[TylerBarnes]

@acao I think they just obscured their /graphql endpoint from the output here. You can see that it's aware of what types are in the remote schema so they definitely have the right url. @peshyyy have you tried lowering options.schema.perPage further? It's hard to say wether or not this is the cause from reading your description but it might help.

[niklaspesch]

Thanks for the clean up @TylerBarnes and yes, I obscured it. Strangely it worked when I changed the GATSBY_CONCURRENT_DOWNLOAD value (to 10) directly in the plugin (fetch-nodes.js). I don't know why the .env file didn't work, but I can keep you updated. I hope this also solves the problem with the crashes that sometimes occur as soon as I set up via Gatsby Cloud.

[TylerBarnes]

@peshyyy can you share your Gatsby Cloud dashboard link? That way i can take a look at your build logs (no one else will be able to visit the link since it's private)

[niklaspesch]

https://www.gatsbyjs.com/dashboard/organization/1d6f1572-f219-468b-8564-c3fb72e5cf3c/sites

[TylerBarnes]

@peshyyy , I'll check out your site but the work this issue is related to was released last week in v4.0.0. Concurrency is now split out into 2 options (1 for files and 1 for GraphQL requests). See here for more info https://github.com/gatsbyjs/gatsby-source-wordpress-experimental/blob/master/CHANGELOG.md#breaking-changes