Since adding a static asset, HMR fails and development mode stops working

kripod commented 6 years ago

Using Windows 10 x64 with Node 8, I encounter the issue below quite often while trying to change a file in development mode (gatsby develop). I'm not sure about the cause, but the issue started when I added an asset to the project's /static directory.

 I  Your site is running at http://localhost:8000
 I  Your graphql debugger is running at http://localhost:8000/___graphql
info changed file at C:\Development\Projects\mvk-web\src\pages\gallery.jsx
 WAIT  Compiling...                                                                 00:16:17
error Plugin dev-404-page returned an error

  Error: EPERM: operation not permitted, open 'C:\Development\Projects\mvk-web\.cache\dev-40  4-page.js'

  - copy-file-sync.js:23 copyFileSync
    [mvk-web]/[fs-extra]/lib/copy-sync/copy-file-sync.js:23:18

  - copy-sync.js:43 Object.copySync
    [mvk-web]/[fs-extra]/lib/copy-sync/copy-sync.js:43:5

  - gatsby-node.js:34 _callee$
    [mvk-web]/[gatsby]/dist/internal-plugins/dev-404-page/gatsby-node.js:34:18

  - gatsby-node.js:51 Object.createPages
    [mvk-web]/[gatsby]/dist/internal-plugins/dev-404-page/gatsby-node.js:51:18

  - api-runner-node.js:104 runAPI
    [mvk-web]/[gatsby]/dist/utils/api-runner-node.js:104:36

  - api-runner-node.js:178
    [mvk-web]/[gatsby]/dist/utils/api-runner-node.js:178:33

  - map.js:27
    [mvk-web]/[gatsby]/[async]/internal/map.js:27:9

  - eachOfLimit.js:64 replenish
    [mvk-web]/[gatsby]/[async]/internal/eachOfLimit.js:64:17

 ERROR  Failed to compile with 1 errors                                             00:16:18
 error  in ./.cache/dev-404-page.js

Module build failed: Error: ENOENT: no such file or directory, open 'C:\Development\Projects\mvk-web\.cache\dev-404-page.js'

 @ ./.cache/sync-requires.js 11:53-123

info changed file at C:\Development\Projects\mvk-web\src\pages\gallery.jsx
 WAIT  Compiling...                                                                 00:16:22
error There was a problem reading the file: C:/Development/Projects/mvk-web/.cache/dev-404-page.js

  Error: ENOENT: no such file or directory, open 'C:\Development\Projects\mvk-web\.cache\dev  -404-page.js'

 ERROR  Failed to compile with 1 errors                                             00:16:23
 error  in ./.cache/dev-404-page.js

Module build failed: Error: ENOENT: no such file or directory, open 'C:\Development\Projects\mvk-web\.cache\dev-404-page.js'

 @ ./.cache/sync-requires.js 11:53-123

Done in 94.08s.

KyleAMathews commented 6 years ago

Does the problem fix itself if you empty the static directory?

KyleAMathews commented 6 years ago

Oh this could be the problem https://github.com/jprichardson/node-fs-extra/issues/320

KyleAMathews commented 6 years ago

Yeah I think we need to add the full path to the copy code here: https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/utils/copy-static-directory.js

KyleAMathews commented 6 years ago

Could you test this and put together a PR if this fixes things?

kripod commented 6 years ago

Unfortunately, I’m currently not able to test it, but will do so as soon as possible. The project reproducing the issue can be found here.

resir014 commented 6 years ago

Having the same issue here, running Windows 10 x64.

kripod commented 6 years ago

@KyleAMathews Unfortunately, your solution didn't work, but I figured out that the root cause of the problem is located here: https://github.com/gatsbyjs/gatsby/blob/524ff171cd890b79ce9535d6dc10550c70365d33/packages/gatsby/src/internal-plugins/dev-404-page/gatsby-node.js#L11

Also, the following error message is shown for a tiny fraction of time on the frontend:

KyleAMathews commented 6 years ago

@kripod yeah it looks like newPath isn't being generated correctly — looks like it has a space somehow in it — could you debug why that is?

kripod commented 6 years ago

@KyleAMathews Actually, I just put some console.log instructions inside node_modules/gatsby/dist/internal-plugins/dev-404-page/gatsby-node.js, and it resulted in the following output:

info changed file at C:\Development\Projects\mvk-web\src\pages\gallery.jsx
 WAIT  Compiling...                                                                 01:01:33

currentPath: C:\Development\Projects\mvk-web\node_modules\gatsby\dist\internal-plugins\dev-404-page\raw_dev-404-page.js
newPath: C:\Development\Projects\mvk-web\.cache\dev-404-page.js
error Plugin dev-404-page returned an error

  Error: EPERM: operation not permitted, open 'C:\Development\Projects\mvk-web\.cache\dev-40  4-page.js'

  - copy-file-sync.js:23 copyFileSync
    [mvk-web]/[fs-extra]/lib/copy-sync/copy-file-sync.js:23:18

>

So, I think there's no weird space character in any of those variables.

KyleAMathews commented 6 years ago

There's two spaces in dev-40 4-page.js where it should be dev-404-page.js

kripod commented 6 years ago

Not sure how, because as you can see, I did the logging straight inside the aforementioned gatsby-node.js file, and I think the file copying operation happens just after logging.

KyleAMathews commented 6 years ago

I'm confused :-) I'm just looking at what you put into the issue. I can't replicate what you're seeing.

kripod commented 6 years ago

I can reproduce the issue using the following procedure:

Clone https://github.com/simonyiszk/mvk-web and run yarn
Run yarn develop
Navigate to http://localhost:8000/teams
Edit src/pages/teams.jsx, repeat until the issue reveals itself

KyleAMathews commented 6 years ago

This is on windows? Can you replicate this on other systems?

kripod commented 6 years ago

The issue happens since implementing i18n support based on the following guide: https://www.gatsbyjs.org/blog/2017-10-17-building-i18n-with-gatsby/

And yes, I'm on Windows 10, haven't tried replicating on other systems yet.

kripod commented 6 years ago

Oh, by the way, the additional 2 spaces were because of console output formatting (wrapping the remaining string to be on a new line with 2 spaces of padding).

Michael-Brooks commented 6 years ago

I too am getting this issue when upgrading to versions 1.9.110 and 1.9.112, but it doesn't happen with my current version of 1.9.45. I haven't tried any versions in between, but thought I would give the official starter version a go and the latest to compare.

szhshp commented 6 years ago

Got same issue, cleaned .cache and public/static which does no help. Win10 64b

szhshp commented 6 years ago

For my scenario, seems the file dev-404-page.js is not generated correctly. It throws an error when changing the file. Create it in .cache manually could be a temporary solution.

KyleAMathews commented 6 years ago

@szhielelp @Michael-Brooks the error seems to be windows specific — could you try debugging what's happening when the file copy fails? Would love a PR fixing things!

kildareflare commented 6 years ago

I'm also seeing this repeatably. I have added some code to gatsby-node to download files from remote urls in order to process them. This all appears to work fine. However intermixed within the file results is an error related to the dev 404 page. It appears that this file is created multiple times and the first few pass, when it fails I've narrowed it down to this line in fs-extra/copy-file-sync.js:

const fdw = fs.openSync(destFile, 'w', stat.mode)

The logs preceding this spit out

newPath C:\Code\Freelance\twobobs_gatsby\.cache\dev-404-page.js
fs-extra/copyFileSync
srcFile: C:\Code\Freelance\twobobs_gatsby\node_modules\gatsby\dist\internal-plugins\dev-404-page\raw_dev-404-page.js
destFile: C:\Code\Freelance\twobobs_gatsby\.cache\dev-404-page.js

But there is clearly a space coming in from somwhere as this is the error in the console

 EPERM: operation not permitted, open 'C:\Code\Freelance\twobobs_gatsby\  .cache\dev-404-page.js'

I'm also on Windows 10 64. node 6.11.3 fs-extra 4.02 full results here: https://gist.github.com/kildareflare/754fee0e2c8002b28afb16464504981f

I've seen this (or similar) in the past, but right now it is only occurring when I try to process the images. If I remove the file download code (below) from onCreateNode, I don't see it.

   console.log(`node.internal.owner: ${node.internal.owner}: ${node.type}`)

    let slideshow = node.data.slideshow;

    if(slideshow && slideshow.length > 0) {
      console.log(`download images: () ${slideshow.length})`);

      //pull down all images and store locally
      slideshow.forEach( (image) => {
        createRemoteFileNode({
          url: image.photo.url,
          store,
          cache,
          createNode
        }).then((result) => {
          console.log(`FILE RESULT ===================================================================================`)
          console.log(result)
        })
      });
      //then we need to process eachimage and replace the original with the new processed image html
    }
    }

I'll keep digging...

Question Should dev-404 be created more than once?

kildareflare commented 6 years ago

I have the same issue. also Windows 10 x64. Node 6.11.3.

I've traced it through and found the line causing the issue is

const fdw = fs.openSync(destFile, 'w', stat.mode)

in fs-extra/copy-file-sync.js.

I've seen this error before, but currently only getting it when running this code in onCreateNode.

      //pull down all images and store locally
      slideshow.forEach( (image) => {
        createRemoteFileNode({
          url: image.photo.url,
          store,
          cache,
          createNode
        }).then((result) => {
          console.log(`FILE RESULT ===================================================================================`)
          console.log(result)
        })
      });
      //then we need to process eachimage and replace the original with the new processed image html
    }

Full error log here: https://gist.github.com/kildareflare/754fee0e2c8002b28afb16464504981f

Log preceding the error spit out the following, which look OK

srcFile: C:\Code\twobobs_gatsby\node_modules\gatsby\dist\internal-plugins\dev-404-page\raw_dev-404-page.js
destFile: C:\Code\twobobs_gatsby\.cache\dev-404-page.js
destFile C:\Code\twobobs_gatsby\.cache\dev-404-page.js

But, the error message defeintly shows some extra spaces:

EPERM: operation not permitted, open 'C:\Code\twobobs_gatsby\  .cache\dev-404-page.js'

I've also noticed that when the error coccurs, the dev-404 page is created multiple times (see full log). Whereas on runs that pass with no errors, the page is only created once.

Using fs-extra 4.0.2 Will keep digging.

kildareflare commented 6 years ago

Updated to node 8.9 and now get a bit more error detail

ERROR  Failed to compile with 1 errors14:42:36

This dependency was not found:

* C:\Code\twobobs_gatsby\.cache\dev-404-page.js in ./.cache/sync-requires.js
To install it, you can run: npm install --save C:\Code\twobobs_gatsby\.cache\dev-404-page.js

But that said, I've now noticed that lots of my error messages have weird gaps in them.

 Invariant Violation: Encountered an error trying to infer a GraphQL typ  e for: "data.slideshow.localFile___NODE". There is no corresponding node with   the id field matching: "C:/Code/twobobs_gatsby/.cache/gatsby-source-  filesystem/f81b92fa02888e16aae3362f6dca8d8d.jpg absPath of file"

Magneticmagnum commented 6 years ago

I'm also running into this problem on a Windows 10 x64 machine :(

Magneticmagnum commented 6 years ago

Forgot to add: Emptying images I use in the project from the static folder will compile correctly. I also have other files in my static folder that I don't import in my pages.

kripod commented 6 years ago

Unfortunately, I'm still getting errors even after removing the static directory, cleaning the cache and re-donwloading node_modules.

kripod commented 6 years ago

I just started a new Gatsby project and observed that the wrong behavior starts when I add an imageSharp object to a GraphQL query. I think that the bug might be related to gatsby-plugin-sharp.

resir014 commented 6 years ago

@kripod Oddly enough, the problem exists for me even without gatsby-plugin-sharp.

MaralS commented 6 years ago

I'm facing to to same problem unfortunately. Anyone can help ?

kripod commented 6 years ago

It turns out that this only happens when I have a browser window with a Gatsby-served page open. When I close my browser and edit a file of my Gatsby project, compilation goes fine. I think that the issue should be related to a non-existing resource being requested by the browser.

MaralS commented 6 years ago

@kripod when I runyarn develop and I'm editing index.js file (in the root folder or in a sub-folder) I still have this error. erreur-compilation

resir014 commented 6 years ago

@kripod Oddly enough, that didn't work.

I managed to get it working again after editing and saving any non-JS file (e.g. markdown) while the server is still running. Although that comes with a caveat that it doesn't really happen immediately. You gotta try doing it to one or two different files, until it gets kicked back up and running again.

This is really weird, and ~~we haven't even scratched the surface on what's actually causing it. /:~~ On second thought, it's most likely the weirdly-added spaces on the error logs that @kildareflare and others have been pointing out, but still...

gaearon commented 6 years ago

I am editing a blog post for the React website, and I see this error on every second edit:

ERROR  Failed to compile with 1 errors                                 19:34:46

 error  in ./.cache/json/blog-2015-09-14-community-roundup-27-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)

 @ ./.cache/sync-requires.js 28:48753-48851

It's significantly disruptive to the workflow. Is this the same issue? How can I help debugging it?

KyleAMathews commented 6 years ago

@gaearon just started to work on this actually :-) Issue for this particular problem @ https://github.com/gatsbyjs/gatsby/issues/3094

KyleAMathews commented 6 years ago

@gaearon a fix is out for the JSON input #3218

gaearon commented 6 years ago

Nice, thanks!

coulson84 commented 6 years ago

EDIT - After using this for a while it still broke with the same EPERM issue so it looks like it reduces the chance of it happening but does not fix it.

I managed to reproduce this issue pretty reliably by doing the following

deleting the .cache folder
npm run develop
make a change to a graphql query on one of my pages (in my instance it was for gatsby-image-plugin but it seems other people have had a similar issue with other plugins and I don't think the plugins have much to do with the issue)

Result `error Plugin dev-404-page returned an error

Error: EPERM: operation not permitted, open 'C:\Users\joe\Personal\jessiecouls onphotography.cache\dev-404-page.js'`

I played around with using the async version of fs.copy instead of the synchronous version and it appears to fix the issue. Frustratingly that means I don't know the real cause or why this fixes the issue because I would expect fs.copy to run the same code as fs.copySync except in a non-blocking manner. It also means I can't write any tests to ensure it is fixed and doesn't regress in a future version. :(

The fix is in the master branch of my fork at https://github.com/coulson84/gatsby if anyone else has been having this issue I would appreciate if you can pull and build this to see if it fixes things for you (N.B. I was getting a failed test from packages\gatsby-transformer-documentationjs\src__tests__\gatsby-node.js prior to making any change to my master branch).

Alternatively if you are feeling lazy you might be able to just copy the below directly in to node_modules/gatsby/dist/internal-plugins/dev-404-page/gatsby-node.js

"use strict";

var _asyncToGenerator2 = require("babel-runtime/helpers/asyncToGenerator");

var _asyncToGenerator3 = _interopRequireDefault(_asyncToGenerator2);

function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }

const path = require(`path`);
const fs = require(`fs-extra`);

exports.createPages = (() => {
  var _ref = (0, _asyncToGenerator3.default)(function* ({ store, boundActionCreators }) {
    if (process.env.NODE_ENV !== `production`) {
      const { program } = store.getState();
      const { createPage } = boundActionCreators;
      const currentPath = path.join(__dirname, `./raw_dev-404-page.js`);
      const newPath = path.join(program.directory, `.cache`, `dev-404-page.js`);

      yield new Promise(function (resolve, reject) {
        fs.copy(currentPath, newPath, err => err ? reject(err) : resolve());
      }).then(function () {
        return createPage({
          component: newPath,
          path: `/dev-404-page/`
        });
      });
    }
  });

  return function (_x) {
    return _ref.apply(this, arguments);
  };
})();
//# sourceMappingURL=gatsby-node.js.map

m-allanson commented 6 years ago

@coulson84 Maybe Gatsby v1.9.138 fixes this issue? It includes #3237, which looks like it landed around the same time as your comment.

coulson84 commented 6 years ago

Unfortunately not - just npm installed 1.9.138 and still getting the same error :(

coulson84 commented 6 years ago

Also on another unsurprising note the code change I mentioned above doesn't solve the problem. It just seems to reduce the chance of the issue showing.

MaralS commented 6 years ago

I tried to fix this issue with @coulson84 method but It didn't work. It's very frustrating. When I encount this error, I have to stop with ctrl + c and run again yarn develop.

KyleAMathews commented 6 years ago

Does this happen on every Gatsby site for y'all? Or just certain ones? Could someone try installing various starters and testing this?

Also, it seems this is a windows-only issue but to double-check, has anyone seen this on Mac?

MaralS commented 6 years ago

@KyleAMathews I will try on my Mac

MaralS commented 6 years ago

@KyleAMathews I confirm that this issue is specific to Windows x64

waywardm commented 6 years ago

Im getting this error all the time on windows 64

ERROR Failed to compile with 1 errors 13:39:49 error in ./.cache/dev-404-page.js

Module build failed: Error: ENOENT: no such file or directory, open 'S:\gatsbysowerby.cache\dev-404-page.js'

@ ./.cache/sync-requires.js 15:53-106

info changed file at S:\gatsbysowerby\src\templates\business-listing.js WAIT Compiling... 13:39:52 error There was a problem reading the file: S:/gatsbysowerby/.cache/dev-404-page.js

Error: ENOENT: no such file or directory, open 'S:\gatsbysowerby.cache\dev-404-page.js'

ERROR Failed to compile with 1 errors 13:39:52 error in ./.cache/dev-404-page.js

waywardm commented 6 years ago

Error: EPERM: operation not permitted, open 'S:\gatsbysowerby.cache\dev-404-page.js'

copy-file-sync.js:23 copyFileSync [gatsbysowerby]/[fs-extra]/lib/copy-sync/copy-file-sync.js:23:18
copy-sync.js:43 Object.copySync [gatsbysowerby]/[fs-extra]/lib/copy-sync/copy-sync.js:43:5
gatsby-node.js:34 _callee$ [gatsbysowerby]/[gatsby]/dist/internal-plugins/dev-404-page/gatsby-node.js:34:18
gatsby-node.js:51 Object.createPages [gatsbysowerby]/[gatsby]/dist/internal-plugins/dev-404-page/gatsby-node.js:51:18
api-runner-node.js:104 runAPI [gatsbysowerby]/[gatsby]/dist/utils/api-runner-node.js:104:36
api-runner-node.js:178 [gatsbysowerby]/[gatsby]/dist/utils/api-runner-node.js:178:33
map.js:27 [gatsbysowerby]/[gatsby]/[async]/internal/map.js:27:9
eachOfLimit.js:64 replenish [gatsbysowerby]/[gatsby]/[async]/internal/eachOfLimit.js:64:17
eachOfLimit.js:49 iterateeCallback [gatsbysowerby]/[gatsby]/[async]/internal/eachOfLimit.js:49:17
onlyOnce.js:12 [gatsbysowerby]/[gatsby]/[async]/internal/onlyOnce.js:12:16
map.js:29 [gatsbysowerby]/[gatsby]/[async]/internal/map.js:29:13
util.js:16 tryCatcher [gatsbysowerby]/[bluebird]/js/release/util.js:16:23
nodeify.js:23 Promise.successAdapter [gatsbysowerby]/[bluebird]/js/release/nodeify.js:23:30
promise.js:566 Promise._settlePromise [gatsbysowerby]/[bluebird]/js/release/promise.js:566:21

ERROR Failed to compile with 1 errors 13:50:05 error in ./.cache/dev-404-page.js

Module build failed: Error: ENOENT: no such file or directory, open 'S:\gatsbysowerby.cache\dev-404-page.js'

kripod commented 6 years ago

@KyleAMathews Is there a chance this could get fixed by v2? Are there any plans to refactor the HMR code, or should we keep investigating the issue even before the release of v2?

KyleAMathews commented 6 years ago

@kripod would love to get this fixed but we still don't understand the underlying cause of the problem. It isn't something that needs to wait for v2. If you could investigate more to discover what's going wrong on windows, that'd be great!

kripod commented 6 years ago

I have already tried to find the reason once, but unfortunately, I couldn't. It's good to know that we should not expect a fix by v2, though.

krachim commented 6 years ago

Hi there,

i have the same error on Windows x64. I tried to figure out why, but got no more time. My findings so far: I guess it´s kind of a timing problem. When add a Filter to check if dest file is there in
./node_modules/gatsby/dist/internal-plugins/dev-404-page/gatsby-node.js:

const filterFunc = (src, dest) => {
  fs.access( dest, fs.constants.R_OK | fs.constants.W_OK, err => {
      if (err) {
        console.log(err)
        return false
      } else {
        return true
      }
    }
  )
}

fs.copySync(currentPath, newPath, { filter: filterFunc })

it seems to work without any error occuring.

another quick and dirty workarround is to change copySync to copy. fs.copySync(currentPath, newPath); in fs.copy(currentPath, newPath);

both seems to work for me ...

Hope this helps someone.
Have a nice Day!

gatsbyjs / gatsby

Since adding a static asset, HMR fails and development mode stops working #2819