gatsby-wordpress-source image localFile null / if i add new Image somewhere on the page another Posts Image localFile becomes null

[arturhenryy]

Description

I have a several custom post types which have either an repeater for images or a featured image. Until a random lower amount of posts ~20 everything seems fine but than if i add new images to the page like for example in my projects post, the localfile for another post not related to the one becomes null. i could not detect any pattern for that. it seems random and looks like there is just certain space / limit available and if that space is full new localFile items override the old ones. i tried deleting .cache and public and reuploading the images but no effect.

when i check the api response from wordpress everything is fine and the image sources are all in the api response

i first thought that the problem was when i upload a image which is smaller in its natural dimensions than defined in the graphql query like this.

project_images {
          image {
            localFile {
              childImageSharp {
                fixed(width: 940) {
                  src
                  width
                  height
                  originalName
                }
              }
            }
          }

e. g i upload an image with 900px width. if i do so localFile will be null.

I will get this in the terminal during build Generating image thumbnails [============================ ] 14/16 0.5 secs 100% The requested width "940px" for a resolutions field for the file /Users/arturnagenrauft/dev/si-page/.cache/gatsby-source-filesystem/0a7d60e9c09b0ab319d3d49daa855b4f.jpg was larger than the actual image width of 900px! If possible, replace the current image with a larger one.

but i reduced the fixed width so the images are not to big for the required width but i still get this issue

Expected result

I would expect null for entries which have no images but in this case the thumbnail should still get generated and in the localFile be populated.

Actual result

TypeError: Cannot read property 'localFile' of null

localFile seems to get deleted by another localFile

Environment

gatsby-config.js

const config = require('./data/SiteConfig')

const pathPrefix = config.pathPrefix === '/' ? '' : config.pathPrefix

module.exports = {
  pathPrefix: config.pathPrefix,
  siteMetadata: {
    siteUrl: config.siteUrl + pathPrefix,
    rssMetadata: {
      site_url: config.siteUrl + pathPrefix,
      feed_url: config.siteUrl + pathPrefix + config.siteRss,
      title: config.siteTitle,
      description: config.siteDescription,
      image_url: `${config.siteUrl + pathPrefix}/logos/logo-512.png`,
      author: config.userName,
      copyright: config.copyright
    }
  },
  plugins: [
    'gatsby-plugin-sass',
    'gatsby-plugin-react-helmet',
    //'gatsby-plugin-page-transitions',
    {
      resolve: 'gatsby-source-wordpress',
      options: {
        // The base url to your WP site.
        baseUrl: 'si.mobydigg.de',
        // baseUrl: 'wpgatsby.wtf',
        // WP.com sites set to true, WP.org set to false
        hostingWPCOM: false,
        // The protocol. This can be http or https.
        protocol: 'http',
        // Use 'Advanced Custom Fields' Wordpress plugin
        useACF: true,
        auth: {},
        // Set to true to debug endpoints on 'gatsby build'
        verboseOutput: true,
        includedRoutes: [
          "/*/*/categories",
          "/*/*/posts",
          "/*/*/projects",
          "/*/*/jobs",
          "/*/*/pages",
          "/*/*/media",
          "/*/*/tags",
          "/*/*/taxonomies",
          "/*/*/menus"
        ],
      }
    },
    'gatsby-transformer-sharp',
    'gatsby-plugin-sharp',
    // {
    //   resolve: 'gatsby-plugin-google-analytics',
    //   options: {
    //     trackingId: config.googleAnalyticsID,
    //   },
    // },
    {
      resolve: 'gatsby-plugin-nprogress',
      options: {
        color: config.themeColor
      }
    },
    'gatsby-plugin-catch-links',
    'gatsby-plugin-sitemap',
    {
      resolve: 'gatsby-plugin-manifest',
      options: {
        name: config.siteTitle,
        short_name: config.siteTitle,
        description: config.siteDescription,
        start_url: config.pathPrefix,
        background_color: config.backgroundColor,
        theme_color: config.themeColor,
        display: 'minimal-ui',
        icons: [
          {
            src: '/logos/logo-192x192.png',
            sizes: '192x192',
            type: 'image/png'
          },
          {
            src: '/logos/logo-512x512.png',
            sizes: '512x512',
            type: 'image/png'
          }
        ]
      }
    },
    'gatsby-plugin-offline'
  ]
}

System: OS: macOS High Sierra 10.13.2 CPU: (4) x64 Intel(R) Core(TM) i5-7267U CPU @ 3.10GHz Shell: 5.3 - /bin/zsh Binaries: Node: 8.12.0 - /usr/local/bin/node Yarn: 1.12.3 - /usr/local/bin/yarn npm: 6.4.1 - /usr/local/bin/npm Browsers: Chrome: 71.0.3578.80 Firefox: 63.0 Safari: 11.0.2 npmPackages: gatsby: ^2.0.63 => 2.0.63 gatsby-image: ^2.0.22 => 2.0.22 gatsby-link: ^2.0.7 => 2.0.7 gatsby-plugin-catch-links: ^2.0.9 => 2.0.9 gatsby-plugin-feed: ^2.0.11 => 2.0.11 gatsby-plugin-google-analytics: ^2.0.8 => 2.0.8 gatsby-plugin-manifest: ^2.0.11 => 2.0.11 gatsby-plugin-nprogress: ^2.0.7 => 2.0.7 gatsby-plugin-offline: ^2.0.18 => 2.0.18 gatsby-plugin-page-transitions: ^1.0.7 => 1.0.7 gatsby-plugin-react-helmet: ^3.0.4 => 3.0.4 gatsby-plugin-sass: ^2.0.7 => 2.0.7 gatsby-plugin-sharp: ^2.0.14 => 2.0.14 gatsby-plugin-sitemap: ^2.0.3 => 2.0.3 gatsby-plugin-styled-components: ^3.0.4 => 3.0.4 gatsby-plugin-twitter: ^2.0.8 => 2.0.8 gatsby-remark-autolink-headers: ^2.0.12 => 2.0.12 gatsby-remark-copy-linked-files: ^2.0.7 => 2.0.7 gatsby-remark-images: ^3.0.1 => 3.0.1 gatsby-remark-prismjs: ^3.1.4 => 3.1.4 gatsby-remark-responsive-iframe: ^2.0.7 => 2.0.7 gatsby-source-filesystem: ^2.0.11 => 2.0.11 gatsby-source-wordpress: ^3.0.19 => 3.0.19 gatsby-transformer-remark: ^2.1.15 => 2.1.15 gatsby-transformer-sharp: ^2.1.9 => 2.1.9 npmGlobalPackages: gatsby-cli: 2.4.5

[arturhenryy]

this is so weird. everytime i run yarn dev the amount of thumbnails generated varies.

Generating image thumbnails [==============================] 244/244 9.0 secs 100%

Generating image thumbnails [==============================] 286/286 9.8 secs 100%

Generating image thumbnails [==============================] 234/234 8.5 secs 100%

Generating image thumbnails [==============================] 334/334 11.9 secs 100%

[arturhenryy]

@pieh i have seen you developed the gatsby image plugins. have you encountered something like this before?

[pieh]

I didn't see that, I'll see if I can reproduce it

[arturhenryy]

@pieh if it helps i could send you the repo

[btk]

I have been pulling my hair over this for the last 2 hours. Seems like this is new, here is my graphql return if it helps;

  query {
    wordpressWpMedia {
      id
      modified
      source_url
      media_details {
        width
        height
        file
      }
      localFile {
        childImageSharp {
          resolutions(width: 674, height: 350, quality: 80, cropFocus: CENTER) {
            src
          }
        }
      }
    }
  }

"wordpressWpMedia": {
      "id": "77b886e3-b3c2-537d-b953-a18692782068",
      "modified": "2018-12-09T22:37:04.000Z",
      "source_url": "https://example.com/wp-content/uploads/2018/12/xxx.jpg",
      "media_details": {
        "width": 700,
        "height": 465,
        "file": "2018/12/xxx.jpg"
      },
      "localFile": null
    }

Seems like it is just not downloading or rendering the new files, I will see if I can create a repo.

[arturhenryy]

@btk thanks for looking into this. yeah i know seems to be really weird. i mean everytime i run yarn dev i get different results. different localfiles missing and others populated again. sometimes all are generated. but that happens rarely. but yeah so basically you could reproduce this problem? getting really desperate because we have to finish our app till friday. and this problem only occured now since we replaced our demo content with the bigger client data set (stupid mistake doing this in the end haha in hindsight)

[pieh]

@arturhenryy yes, please share repo if you can (either here or DM me on twitter - @mipiechowiak if you prefer to keep it private)

[pieh]

Ok, I can definitely reproduce this it with repo from @arturhenryy

[btk]

@pieh my problem was about a storage trouble about me. So don't mind me, it might not be a new problem.

[pieh]

After some debugging - seems like server is getting overloaded and returns 503 for bunch of images, but it's getting silently ignored leaving some wordpress media nodes without localFile. Defenitely we need some kind of rate limiter and retry system here

[arturhenryy]

@pieh do you mean our wp server throws 503´s for some images? is there anything like a quick fix we can do about this? like migrating to a stronger server system in the mean time? i mean worstcase would be switching back to normal img tags and just using the src of the response. do you know how long it will take to fix this issue approximately? thank you so much for looking into this issue. really appreciate it.

[arturhenryy]

@pieh hey i just wanted to check if there is some progress with this issue. we upgraded the server and at first sight it fixed the issue temporally but adding more images led to missing images across the page again

[arturhenryy]

@pieh @sidharthachatterjee hey guys we are about to setting up another site but it will also be pretty image heavy. and i really want to use gatsby so i wanted to ask if this error will be fixed at all or do i have to look for another backend then wordpress here instead?

[sidharthachatterjee]

Hey @arturhenryy

Could you share the reproduction repo with me as well please? I'm at https://twitter.com/chatsidhartha

On a side note, based on what @pieh said above, since this is a rate limiting issue, a simple fix could be to whitelist the IP or domain that the build is occurring in

[arturhenryy]

@sidharthachatterjee thanks for the reply i sent you the repo via twitter.

ok could you explain this in a bit more detail please? where would i need to whitelist the ip?

[sidharthachatterjee]

@arturhenryy Depends on where the WordPress instance is hosted, I suppose

Typically this is in the nginx or Apache config that sits in front of the WordPress installation

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[arturhenryy]

not stale

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

Thanks again for being part of the Gatsby community!

[hesselberg]

Recently ran into this issue as well. I'll try whitelisting IP addresses and see if it solves the issues.

(not stale)

[arturhenryy]

@hesselberg i tried whitelisting ip´s but it did not help, so you can save your time here.

[arturhenryy]

@sidharthachatterjee can you please reopen this ticket as it is not resolved yet and i am not able to attach the not stale label here

[hesselberg]

@arturhenryy Just discovered it wasn't actually related to this. I ended up deleting a couple of Wordpress pages where the issue was, then rebuilt them, then it worked.

Cannot really say why it happened, but everything seems to work fine now. I still think this issue in general is valid, but at least my problem was solved by removing the Wordpress pages, and then recreating them.

[kristojorg]

I am seeing a similar issue with gatsby-source-shopify. If anybody had any luck resolving I would love to know. Not sure if it is rate limited APIs or bad internet or some internal problem.

[arturhenryy]

@hesselberg that might be a temporary fix, but at the next build you could encounter the same issues again. at least thats how it was with our page. so basically what i do right now is building countless times until i get a build with all necessary images haha.

[hesselberg]

@arturhenryy I see. And I agree, I ran into the same issue now. Hmmpf. Not good.

[arturhenryy]

@sidharthachatterjee could you please reopen this ticket?

[kristojorg]

I believe the source library needs to wrap the call to createRemoteFileNode in a try catch block if they want to handle the case of that function failing. I tried this with a local copy of source-shopify and it did work. I was able to retry the download and also throw a console error or else fail more gracefully

[amykapernick]

I wonder whether something has happened recently, because I've also been having this issue and unable to resolve in the last few days and it looks like a lot of people are having the same issue.

[kristojorg]

If so, I think it would be in the gatsby-source-filesystem createRemoteFileNode API. That is what is responsible for downloading the remote images, and what is returning null when it fails for whatever reason. We should discuss what a better option would be than returning null. I tried wrapping it in a try catch block, which is catching the error case. I suggest source packages wrap it in try catch and 1. log an error to the console that the image did not download, and 2. skip creating the node, while possibly providing a config for a number of retries to allow.

[hesselberg]

@kristojorg do you possibly have a gist or a PR? I'm quite new to Gatsby let alone the source packages. Would definitely prefer if it would retry. It appears to work if you keep rebuilding, although I don't have any specific pattern.

[kristojorg]

I don't have a PR ready yet, and I'm working on the source-shopify. However if I get that figured out today or tomorrow I can try to quickly do the same for the wordpress source and at least you'll have a starting point : )

[hesselberg]

@kristojorg great! Hope you succeed :-)

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

Thanks again for being part of the Gatsby community!

[amykapernick]

Is someone able to re-open this, there's definitely still activity and this has been tagged as confirmed

[hesselberg]

@kristojorg

I don't have a PR ready yet, and I'm working on the source-shopify. However if I get that figured out today or tomorrow I can try to quickly do the same for the wordpress source and at least you'll have a starting point : )

Did you ever get around to finding a good solution?

[kristojorg]

No I ended up moving back to somewhere with good enough internet where I stopped seeing this issue. There should be a flag such that in development if downloading the localFile fails, the entire node will be skipped, and in production the build will retry download and then fail as a last resort.

[kristojorg]

In the gatsby docs this is the example usage of createRemoteFileNode:

exports.downloadMediaFiles = ({
  nodes,
  store,
  cache,
  createNode,
  createNodeId,
  _auth,
}) => {
  nodes.map(async node => {
    let fileNode
    // Ensures we are only processing Media Files
    // `wordpress__wp_media` is the media file type name for Wordpress
    if (node.__type === `wordpress__wp_media`) {
      try {
        fileNode = await createRemoteFileNode({
          url: node.source_url,
          parentNodeId: node.id,
          store,
          cache,
          createNode,
          createNodeId,
          auth: _auth,
        })
      } catch (e) {
        // Ignore
      }
    }

    // Adds a field `localFile` to the node
    // ___NODE appendix tells Gatsby that this field will link to another node
    if (fileNode) {
      node.localFile___NODE = fileNode.id
    }
  })
}

The catch block is where source-shopify or source-wordpress needs to either fail in production or delete the node in development

[arturhenryy]

@kristojorg you will probably still encounter this issue when you add more images.

yeah i think implementing something like this could do the trick

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options)
            .then(function(result) {
                /* on success */
                resolve(result);
            })
            .catch(function(error) {
                /* on failure */
                fetch_retry(url, options, n - 1) // <--- leap of faith, this will just work magically! Don't worry!
                    .then(/* one of the remaining (n - 1) fetch succeed */)
                    .catch(/* remaining (n - 1) fetch failed */);
            })
    });
}

[kristojorg]

@arturhenryy it might actually make sense to use a lib like this https://github.com/zeit/async-retry to manage retry logic

[wardpeet]

feel free to create a PR so people can this this. If this actually works we are happy to get this in! 🎉

[rexsuit]

Has anybody with an understanding of how gatsby fetches data gotten any closer to figuring this problem out? gatsby-source-wordpress is basically unusable for me because of this bug :(

[arturhenryy]

@wardpeet would love to help but if i see this right this is a core bug in the functionality of several source plugins which needs a deep understanding of the underlying system because changing stuff like timing or retrying might have several implications for other parts of the system here. so this should be addressed by some gatsby members in my opinion

[kristojorg]

@arturhenryy @orshick @hesselberg here is a draft PR that should fix the issue. I don't have a local WP site to test it on though, so the code has not been run and no tests have been added. If you would like to continue the PR and get it merged, please do so!

https://github.com/gatsbyjs/gatsby/pull/13478

[arturhenryy]

@kristojorg sounds great mate thanks. i will test it.

[violy]

@arturhenryy @orshick @hesselberg here is a draft PR that should fix the issue. I don't have a local WP site to test it on though, so the code has not been run and no tests have been added. If you would like to continue the PR and get it merged, please do so!

13478

yes, maybe my PR https://github.com/gatsbyjs/gatsby/pull/13110 should also solve overload issues

[lauritowal]

Was this fixed? I still notice a similar behavior as mentioned by @arturhenryy :/

[signorekai]

Is there any kind of temporary solution to this issue?

[signorekai]

For anyone who's facing issue with this, a temporary fix is to limit the concurrent image download that GatsbyJS opens.

Example package.json:

{
  ...
  "scripts": {
    "dev": "GATSBY_CONCURRENT_DOWNLOAD=5 gatsby develop"
  }
}

[EricSSartorius]

Seeing this problem now as well, still no fix for it I am guessing? @signorekai has limiting downloads fixed your problem completely?