[fsevents bug] Stuck at "source and transform nodes" / "createPagesStatefully" on MacOS

ehowey commented 5 years ago

Description

I am working on a theme, local build was working fine with no problems, and recently upgraded all the dependencies to Gatsby 2.14.0 and both gatsby develop and gatsby build hang at source and transform nodes in my local dev environment.

Interestingly it works and builds on Netlify. This would point to it being something on my system. I have deleted the node modules folders in the workspaces and the root workspace folder and done a fresh yarn command. I also deleted the yarn.lock and package.lock files...not sure if this would cause problems.

Steps to reproduce

Theme repo is here: Gatsby-Theme-Catalyst-Core

Starter repo is here: Gatsby-Starter-Catalyst-Core

I have this setup in a yarn workspace for development however the same issue occurs if you do a fresh install of the starter using gatsby new my-catalyst-starter-core https://github.com/ehowey/gatsby-starter-catalyst-core

Expected result

Build successfully

Actual result

yarn workspace v1.17.3 yarn run v1.17.3 $ gatsby develop success open and validate gatsby-configs - 0.122 s success load plugins - 1.964 s success onPreInit - 0.073 s success initialize cache - 0.056 s success copy gatsby files - 0.242 s success onPreBootstrap - 0.087 s ⠙ source and transform nodes

Environment

System: OS: macOS High Sierra 10.13.6 CPU: (2) x64 Intel(R) Core(TM)2 Duo CPU P8600 @ 2.40GHz Shell: 3.2.57 - /bin/bash Binaries: Node: 12.9.1 - /usr/local/bin/node Yarn: 1.17.3 - /usr/local/bin/yarn npm: 6.11.2 - /usr/local/bin/npm Languages: Python: 2.7.16 - /usr/local/bin/python Browsers: Chrome: 76.0.3809.100 Firefox: 67.0.2 Safari: 12.1.2 npmGlobalPackages: gatsby-cli: 2.7.40

ehowey commented 5 years ago

@Js-Brecht Changing the thread pool size didn't seem to make much difference. I get the following output from the standard gatsby develop command with the changes you mentioned above. Still on the basic hello-world gatsby starter.

Erics-MBP:my-hello-world-starter erichowey$ gatsby develop
success open and validate gatsby-configs - 0.050 s
success load plugins - 0.119 s
success onPreInit - 0.036 s
success initialize cache - 0.050 s
success copy gatsby files - 0.281 s
onPreBootstrap  load-babel-config
success onPreBootstrap - 0.131 s
sourceNodes     internal-data-bridge
success source and transform nodes - 0.105 s
success Add explicit types - 0.030 s
success Add inferred types - 0.186 s
success Processing types - 0.145 s
success building schema - 0.535 s
success createPages - 0.032 s
createPagesStatefully   dev-404-page
onCreatePage    internal-data-bridge
onCreatePage    prod-404
createPagesStatefully   gatsby-plugin-page-creator
createPagesStatefully   gatsby-plugin-page-creator
createPagesStatefully   gatsby-plugin-page-creator
createPagesStatefully   gatsby-plugin-page-creator
createPagesStatefully   gatsby-plugin-page-creator
createPagesStatefully   gatsby-plugin-page-creator
onCreatePage    internal-data-bridge
onCreatePage    prod-404
success createPagesStatefully - 9.098 s
success onPreExtractQueries - 0.030 s
success update schema - 0.129 s
success extract queries from components - 0.336 s
success write out requires - 0.057 s
success write out redirect data - 0.044 s
success onPostBootstrap - 0.045 s
⠀
info bootstrap finished - 16.437 s
⠀
success run static queries - 0.038 s
success run page queries - 0.109 s — 2/2 27.65 queries/second
onCreateWebpackConfig   webpack-theme-component-shadowing
onCreateWebpackConfig   webpack-theme-component-shadowing
success start webpack server - 1.506 s — 1/1 4.63 pages/second
 DONE  Compiled successfully in 4699ms

This is an example where it hangs on createPagesStatefully. I got it to continue the build by resizing the terminal window. What is internal-data-bridge any chance that could be the culprit in some way? That was featuring in both of the commands you have asked me to run so far.

Js-Brecht commented 5 years ago

Can you show what line it hung on?

ehowey commented 5 years ago

createPagesStatefully dev-404-page

I am not 100%sure whether the dev-404-page part was there yet but it hung on the first instance of createPagesStatefully

Js-Brecht commented 5 years ago

Thanks. I'd like to try some more changes now, if you don't mind.

Please update the indicated lines in the following files:

node_modules/gatsby/dist/internal-plugins/internal-data-bridge/gatsby-node.js

- 114   chokidar.watch(pathToGatsbyConfig).on(`change`, () => {
+ 114   chokidar.watch(pathToGatsbyConfig, { useFsEvents: false }).on(`change`, () => {

node_modules/gatsby/dist/internal-plugins/dev-404-page/gatsby-node.js

- 30     chokidar.watch(source).on(`change`, () => copy()).on(`ready`, () => done());
+ 30     chokidar.watch(source, { useFsEvents: false }).on(`change`, () => copy()).on(`ready`, () => done());

node_modules/gatsby-page-utils/dist/watch-directory.js

  26               chokidar.watch(glob, {
  27                 cwd: path,
+ 28                 useFsEvents: false,
  29               }).on("add", function (path) {

node_modules/gatsby/dist/utils/get-static-dir.js

- 51   chokidar.watch(staticDir).on(`add`, path => {
+ 51   chokidar.watch(staticDir, { useFsEvents: false }).on(`add`, path => {

node_modules/gatsby/dist/query/query-watcher.js

- 237   watcher = chokidar.watch([slash(path.join(rootDir, `/src/**/*.{js,jsx,ts,tsx}`)), ...packagePaths]).on(`change`, path => {
+ 237   watcher = chokidar.watch([slash(path.join(rootDir, `/src/**/*.{js,jsx,ts,tsx}`)), ...packagePaths], { useFsEvents: false }).on(`change`, path => {

I have a suspicion that chokidar might be at fault here. It was upgraded to v3.x about a month ago, and it looks like they either started using fsevents, or they did something that caused some errant behavior with relation to fsevents. They have some open issues similar to the ones we are facing here, with node hanging when using chokidar.watch(). This seems to fit, since it is localized to MacOS due to fsevents being a Mac process, and the build seems to be failing in the modules where it is being used, or in modules that are writing out/processing files that are likely being watched.

chokidar.watch() exists in gatsby-source-filesystem, too, and that is where it was failing with your other repo, @ehowey; not to mention, gatsby-source-filesystem is not being called in your minimal repo, which kind of explains why it is making it past source and transform. There are instances of chokidar before internal-data-bridge, but I think on locations that are not affecting the build (e.g. query-watcher). However, I believe internal-data-bridge is the reason it was hanging while running the debugger; and in a straight-through run, it was likely affecting later stages of the build.

Let me know if this fixes the issues, or even if it makes it further than it has been. :crossed_fingers:

ehowey commented 5 years ago

@Js-Brecht You are getting somewhere! I ran gatsby develop probably 15 times. It never hung on source and transform nodes or createPagesStatefully but it seemed to hang, maybe 2/10 times, on start webpack server. I could bump this along with the resizing terminal trick as well. Any chance you missed an instance of chokidar/fsevents that would relate to start webpack server?

As a side note it also seemed to move through the steps of the develop process much faster than before if that has anything to do with it?

Js-Brecht commented 5 years ago

That’s really good to hear. I did leave one instance of chokidar, on purpose, and it is right after it finishes the bootstrap and starts the server. It is in node_modules/gatsby/dist/commands/develop.js towards the end of the startServer() function. I’m not at a computer right now, or I would give you a diff.

You can find the exact line by doing:

cat node_modules/gatsby/dist/commands/develop.js |grep -n ‘chokidar’

I was pretty sure that if this fixed the bootstrap, then it would have to still hang at the server. Let me know if changing that makes it run reliably, and I’ll submit a PR and take care of letting people know.

I’m actually not surprised that it makes it run faster. The build on a barebones project usually takes me maybe a second, and I saw some crazy long runtimes for certain stages in your debug output. fsevents is supposed to speed things up, and take a lot of the load off the cpu, but there’s something funny going on that’s making it get stuck, instead.

ehowey commented 5 years ago

@Js-Brecht My hat goes off to you. I have no idea how you pulled that rabbit out of a hat and bug fixed such a weird one from a distance but good work! I will wait for the diff to make changes to develop.js as I do not want to change the wrong thing, but I suspect that will fix it as it was hanging at that very last step when it starts the server a couple of times.

Js-Brecht commented 5 years ago

Here's the diff:

node_modules/gatsby/dist/commands/develop.js

- 337   chokidar.watch(watchGlobs).on(`change`, async () => {
+ 337   chokidar.watch(watchGlobs, { useFsEvents: false }).on(`change`, async () => {

I really appreciate your help, running all these tests. This has been a tricky one, but it has been an opportunity to dig into the internals of MacOS, and I always enjoy learning new things :slightly_smiling_face:

Please let me know how it goes; not completely out of the woods yet :sweat_smile:.

ehowey commented 5 years ago

Works! I just ran gatsby develop 10+ times and it worked flawlessly. Thanks for your help in looking into this. Hopefully this makes for an improvement for the group of us experiencing this!

Js-Brecht commented 5 years ago

Great! Here in a little while, when I’ve got a couple minutes, I’ll put together a PR. Once it’s done, you’ll be able to use gatsby-dev-cli with my repo so that you have a working dev environment until a patch gets published (if you aren’t familiar with gatsby-dev-cli, I can help). I might try recruiting you to test the changes anyway, since I don’t have the right OS... same goes for everybody else on this thread experiencing this issue.

I’ll post back here when it’s done.

me4502 commented 5 years ago

I've found another separate issue that also causes this symptom - https://github.com/gatsbyjs/gatsby/issues/17935

If anyone is using LokiJS and resizing the terminal doesn't fix it, it's quite likely the issue I've found.

Js-Brecht commented 5 years ago

Hey guys, @ehowey, please check out PR #17938. If anybody is willing to test, please do, and drop a line on the PR.

You can snag my repo, and use it as your gatsby source in your site by using gatsby-dev-cli. First, you need gatsby-dev-cli

# Use your package manager of choice, and install globally
npm install -g gatsby-dev-cli

Then, just clone the repo, and bootstrap it.

git clone --single-branch -b macos-fsevents-fix https://github.com/Js-Brecht/gatsby
cd gatsby
yarn bootstrap
# Set the target repo path for gatsby-dev
gatsby-dev -p "${PWD}"

Then go to the site you wish to use the repo in, and run gatsby-dev

# Disable file watching, unless you intend on making changes to the gatsby repo
gatsby-dev --scan-once

ricard0javier commented 5 years ago

in my case, I'm using OSX and IntelliJ, I had to:

Close the project in IntelliJ
Try again (npm start)
and reopen the project in Intellij I guess that the issue is related with IntelliJ indexing files

Js-Brecht commented 5 years ago

@steinitz, @rheinardkorf, @hbthen3rd, @sharvit, @JaKXz, @emilyaviva, @MaximeHeckel

Any of you guys willing to test #17938?

ehowey commented 5 years ago

I should be able to take a look later tonight when I am home. Thanks for all your work on this! On Oct 1, 2019, 10:23 AM -0600, Jeremy Albright notifications@github.com, wrote:

@steinitz, @rheinardkorf, @hbthen3rd, @sharvit, @JaKXz, @emilyaviva, @MaximeHeckel Any of you guys willing to test #17938? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rishabhaggarwal2 commented 5 years ago

Any update on the fix for this? It's freezing at source and transform nodes for me and my friend as well after it tried [Firebase Source] Fetching data for ...

Resizing the terminal unfortunately doesn't seem to fix it for us

Js-Brecht commented 5 years ago

@rishabhaggarwal2 There's a similar issue that sounds like it might be what you're experiencing, where it will hang while retrieving data from an online source. Can you try using GATSBY_CONCURRENT_DOWNLOAD=10 gatsby develop?

rubengmurray commented 5 years ago

Seeing this too. Can't run gatsby develop locally at all on the lumen startup. Keeps hanging on createPageStatefully or source and transform nodes

macOS Mojave 10.14.6
Gatsby CLI version 2.7.7
Gatsby version 2.14.1

Js-Brecht commented 5 years ago

For anybody else finding this issue, try

CHOKIDAR_USEPOLLING=1 gatsby develop

That should disable fsevents on MacOS, which seems to be a reliable workaround.

steinitz commented 5 years ago

@Js-Brecht I confirm that it seems to fix it more permanently than the other workarounds mentioned here. Thanks for sharing.

Js-Brecht commented 5 years ago

@steinitz you say “more” permanently. Do you mean to say that it still happens when you use that switch?

steinitz commented 5 years ago

@Js-Brecht No, sorry to be unclear.

I was alluding to the fact that other workarounds, including my own, seemed to work for a while but the problem returned. With your solution I provoked it by making changes and re-running gatsby develop (with your enhancement) but the build continues to work.

That said, this has been a rollercoaster ride so I remain emotionally prepared :smile: for the problem to return.

To be clear: so far, so good.

Oops, update: I relaunched WebStorm for a final test before posting this and now it's hung on source and transform nodes in the WebStorm terminal.

Boty22 commented 5 years ago

For anybody else finding this issue, try
CHOKIDAR_USEPOLLING=1 gatsby develop
That should disable fsevents on MacOS, which seems to be a reliable workaround.

@Js-Brecht I'm using ubutu 18.04 and I'm running into the same problem occasionally. Is there any suggestion for my OS? Would you provide some detail about the possible causes for this problem?

ehowey commented 5 years ago

This has been solved thanks to the diligent work of @Js-Brecht and @RomanHotsiy. It was an upstream issue in fsevents, way beyond what I could have figured out on my own, and should be fixed in future updates once these changes are implemented and migrate down from fsevents into gatsby itself. For now a reliable workaround is to resize the terminal window, there is a way to fix this in your repo itself which is discussed here: https://github.com/gatsbyjs/gatsby/pull/17938 however you would have to redo this fix after any changes to your node_modules folder and might not be worth the work depending on how often you update your package versions.

I will leave the issue open until the fix migrates downstream into gatsby itself.

Js-Brecht commented 5 years ago

@Boty22 Ubuntu doesn't use fsevents, so it's probably something different. Some issues have been encountered when retrieving data from a remote location; check out #6654 and #17940 for some details on fixing concurrency issues.

Quick question: does resizing your terminal help? If so, then it could be something similar to this issue.

Boty22 commented 5 years ago

@Js-Brecht resizing the terminal does not help for Ubuntu. I'm retrieving data from an AirTable table using the plugin gatsby-source-airtable BTW

Js-Brecht commented 5 years ago

That could be the problem. Check out this comment. If that doesn't work, might be better to open a new ticket

Js-Brecht commented 5 years ago

@steinitz sorry, I missed your comment. Can you try the fix described in #17938? More specifically, here and here. If it doesn't work, then there's probably more going on in your case.

rubengmurray commented 5 years ago

Haven't had any issues with CHOKIDAR_USEPOLLING=1 gatsby develop since mentioned.

Thanks for the workaround @Js-Brecht

canvaspixels commented 5 years ago

I'm seeing this too with 2.15.28 when I do gatsby build. Do i need to stop the gatsby develop in the other terminal? It's intermittently happening

canvaspixels commented 5 years ago

Happened again without the dev server running. I've got a simple blog from following the blog tutorial though.

canvaspixels commented 5 years ago

It seems to be almost every other run. I'm on a Mac btw

RomanHotsiy commented 5 years ago

@canvaspixels does resizing the terminal window unstuck your build? If so, please try this and let us know if that helped https://github.com/gatsbyjs/gatsby/pull/17938#issuecomment-540661991

canvaspixels commented 5 years ago

@RomanHotsiy that indeed sorts it! Thanks!

Js-Brecht commented 5 years ago

Hey everybody, the patched version of fsevents has been published. You should be able to simply delete your yarn.lock file, and run yarn again. Each of your dependencies should automatically pick up fsevents@2.1.1, which should resolve the issue.

karlhorky commented 5 years ago

Be careful - deleting your whole yarn.lock file can also cause other packages to be upgraded.

Another, more precise option if you're using yarn is to delete just the lines relating to fsevents in the yarn.lock and re-run yarn. This will cause only the affected package to be upgraded. So for example, I removed the following lines:

- 
- fsevents@^2.0.6:
-   version "2.0.7"
-   resolved "https://registry.yarnpkg.com/fsevents/-/fsevents-2.0.7.tgz#382c9b443c6cbac4c57187cdda23aa3bf1ccfc2a"
-   integrity sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==

Another way of doing this is to use the resolutions feature of Yarn (https://yarnpkg.com/lang/en/docs/selective-version-resolutions/). By adding the following to your package.json and then running yarn, it will also upgrade transitive dependencies and update your yarn.lock file:

  "resolutions": {
    "fsevents": "^2.1.1"
  }

After you have done this and your lockfile is updated, you can remove the resolutions property from package.json again.

Edit: removed previous, incorrect version of instructions.

Js-Brecht commented 5 years ago

You can also run yarn upgrade chokidar@latest. That should rebuild the lock file with the correct version of fsevents, without touching anything else

Js-Brecht commented 5 years ago

Oh, wait. That's only if you have chokidar as a direct dependency 🙄 . Forgot. @karlhorky is right

rubengmurray commented 5 years ago

I'm using npm. Deleting node_modules, running npm i --save fsevents@2.1.1 and then npm i did the trick for me. Took 19s to startup and I'm using the gatsby-lumen-starter as a base.

-- Edit:

I actually finished what I was working on, pushed to Netlify and it was completely unable to build because of fsevents (error fsevents@2.1.1: The platform 'linux' is incompatible with this module).

I switched to yarn and had the same issue so I yarn removed fsevents entirely, and now local and netlify are both working...

itdsntwork commented 5 years ago

Got the same issue and could not track the reason. This happen to me both on mac and pc. Could try to relate this to internet speed, however this happened to me also when I was connected to high speed network. What seems to be working for me right now is to add this in my env

GATSBY_CONCURRENT_DOWNLOAD=5

and running using --v8-pool-size=128

thomasthep commented 5 years ago

In my case it seems to be my firewall prompt on osx. If I'm fast enough to allow or deny connections coming from outside, it'll succeed flawlessly. Problem is, the prompt dialog pops up for a split second and disapears as Gatsby continues and then fails hanging on one of the many steps mentioned above. Options:

disable firewall (not having that)
whitelist Gatsby (not consistent across projects and upgrades)
find a way to halt on prompt until user selects option
default bind address to localhost (should not require prompt to accept incoming connections).

Js-Brecht commented 5 years ago

@thomasthep The default bind address for the dev server is localhost already. It doesn't even listen for external connections unless you tell it to use your out-facing IP address (or 0.0.0.0). And even then, it doesn't start the dev server until after the bootstrap / build are done. If it's the bootstrap that is causing the firewall prompt, then it would have more to do with what plugins you are using, because Gatsby does not reach out to the internet in its default state.

There shouldn't even be any connections "coming from outside", generally speaking; even when a plugin is collecting information from an external source, the connection originates from your local host, not from the external source, and that is usually accepted as an "established connection" by most firewalls, in my experience, since most firewalls are configured to accept any outgoing connections.

I could see this happening if your firewall is configured to block processes, rather than just connections. In that case, it would be Node that would need to be whitelisted.

Would need more information to really understand why that's happening to you. It would probably be more effective to open a new ticket, though. This one had to do with fsevents on MacOS causing issues, and that has been addressed, which is why it's closed now. Anything not related to fsevents really should have its own issue, to get the attention it deserves.

jiv-e commented 4 years ago

For the record this also can happen if you have your GraphQL server running in debug mode and it's stopped at a breakpoint.

LpmRaven commented 4 years ago

For the record: This started happening to me when I added gatsby-source-s3-image and the s3 bucket reached over 100 images. It goes through all 145 images at the source and transform nodes stage and then hangs there forever. It's still happening, I've tried the fixes above. Luckily, it works after 5-6 attempts so I am not blocked.

saadnoor commented 4 years ago

Weirdly resizing the terminal window worked for me

calluna18 commented 4 years ago

For me, limiting concurrent downloads as described here seems to help.

I added the following line to my .env file. The default is 200. GATSBY_CONCURRENT_DOWNLOAD=50

Not sure if it solves your problem but maybe it helps someone :)

griessel-jc commented 3 years ago

@rishabhaggarwal2 There's a similar issue that sounds like it might be what you're experiencing, where it will hang while retrieving data from an online source. Can you try using GATSBY_CONCURRENT_DOWNLOAD=10 gatsby develop?

Thank you, this fixed it for me. Since I was pulling a ton of content from a third party website it kept hanging on downloading the content. (97% - so close yet so far)

navneet-kumar commented 3 years ago

I'm using npm. Deleting node_modules, running npm i --save fsevents@2.1.1 and then npm i did the trick for me. Took 19s to startup and I'm using the gatsby-lumen-starter as a base.

-- Edit:

I actually finished what I was working on, pushed to Netlify and it was completely unable to build because of fsevents (error fsevents@2.1.1: The platform 'linux' is incompatible with this module).

I switched to yarn and had the same issue so I yarn removed fsevents entirely, and now local and netlify are both working...

I am having the same issue, it's building on fine on my mac and also on github runner which is a container running ubuntu-latest.

but when i am trying to deploy it on prod which is ubuntu 16.x LTS it stucks at this line -

$ npm ci
Killed    .........] \ reify:fsevents: timing reifyNode:node_modules/webpack-dev-server/node_modules/fsevents Completed in 1053ms

how can we remove the fsevents dependency completely ?

theabdulmateen commented 3 years ago

In my case I had been working with a custom graphql server which seemed to get "socket hang up" error, I discovered this when I was using netlify dev but due to proxy reasons I switched back to gatsby develop.

Restarting the custom server seemed to help.

gatsbyjs / gatsby